2024-06-21T01:32:01.415 INFO:root:teuthology version: 0.0.1.dev325+g8e97141 2024-06-21T01:32:01.415 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi/7765946 --owner scheduled_lflores@folio02 --name lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} --verbose -- /home/teuthworker/archive/lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi/7765946/orig.config.yaml 2024-06-21T01:32:01.448 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-21T01:32:01.527 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi/7765946 branch: squid-release description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: null first_in_suite: false job_id: '7765946' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid-release ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd workunit: branch: squid-release sha1: c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd owner: scheduled_lflores@folio02 priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 5830 sha1: c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: squid-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd targets: smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOwIvd5BRM3W6DGBWCV+r4e+pVrv0C/BEdsSUcXMA4MUHRl1oga3CQYlYx9mkmuzSC9vQxR/ztY5Ag4AeF5sGkw= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdoa9KAtv7csCNK9YVvAFgvtVUbdLTb4DIiekKMXreMEYX5OvayhUHaoAFWJGwLVAIwk3ia1qkwv2+7iLVO1qM= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e9714173de9e92c97e8ef1045d333e96b793454 timestamp: 2024-06-20_17:49:38 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-21T01:32:01.527 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/qa; will attempt to use it 2024-06-21T01:32:01.528 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/qa/tasks 2024-06-21T01:32:01.528 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-21T01:32:01.530 INFO:teuthology.task.internal:Checking packages... 2024-06-21T01:32:01.551 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd' 2024-06-21T01:32:01.551 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-21T01:32:01.551 INFO:teuthology.packaging:ref: None 2024-06-21T01:32:01.551 INFO:teuthology.packaging:tag: None 2024-06-21T01:32:01.551 INFO:teuthology.packaging:branch: squid-release 2024-06-21T01:32:01.551 INFO:teuthology.packaging:sha1: c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:32:01.551 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid-release 2024-06-21T01:32:01.696 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2541.gc65be810 2024-06-21T01:32:01.698 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-21T01:32:01.704 INFO:teuthology.task.internal:no buildpackages task found 2024-06-21T01:32:01.705 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-21T01:32:01.755 INFO:teuthology.task.internal:Saving configuration 2024-06-21T01:32:01.767 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-21T01:32:01.781 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-21T01:32:01.805 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi/7765946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-21 01:27:05.473258', 'locked_by': 'scheduled_lflores@folio02', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOwIvd5BRM3W6DGBWCV+r4e+pVrv0C/BEdsSUcXMA4MUHRl1oga3CQYlYx9mkmuzSC9vQxR/ztY5Ag4AeF5sGkw='} 2024-06-21T01:32:01.823 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi/7765946', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-21 01:27:05.471361', 'locked_by': 'scheduled_lflores@folio02', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdoa9KAtv7csCNK9YVvAFgvtVUbdLTb4DIiekKMXreMEYX5OvayhUHaoAFWJGwLVAIwk3ia1qkwv2+7iLVO1qM='} 2024-06-21T01:32:01.823 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-21T01:32:01.829 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-06-21T01:32:01.829 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-06-21T01:32:01.829 INFO:teuthology.run_tasks:Running task console_log... 2024-06-21T01:32:01.910 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fccb6857ac0>, signals=[15]) 2024-06-21T01:32:01.911 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-21T01:32:01.917 INFO:teuthology.task.internal:Opening connections... 2024-06-21T01:32:01.917 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2024-06-21T01:32:01.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-21T01:32:01.998 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-06-21T01:32:02.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-21T01:32:02.073 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-21T01:32:02.080 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2024-06-21T01:32:02.097 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2024-06-21T01:32:02.098 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream" 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9" 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos" 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora" 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9" 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9" 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon" 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/" 2024-06-21T01:32:02.155 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-21T01:32:02.156 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-21T01:32:02.156 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-21T01:32:02.156 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2024-06-21T01:32:02.207 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-06-21T01:32:02.225 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-06-21T01:32:02.226 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-06-21T01:32:02.282 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-06-21T01:32:02.282 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-06-21T01:32:02.282 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-06-21T01:32:02.282 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-06-21T01:32:02.282 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-06-21T01:32:02.282 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-06-21T01:32:02.282 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-21T01:32:02.282 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-06-21T01:32:02.283 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-06-21T01:32:02.283 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-21T01:32:02.283 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-06-21T01:32:02.283 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-21T01:32:02.283 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-21T01:32:02.283 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-21T01:32:02.283 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-06-21T01:32:02.304 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-21T01:32:02.313 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-21T01:32:02.337 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-21T01:32:02.338 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2024-06-21T01:32:02.340 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-06-21T01:32:02.356 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-21T01:32:02.362 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-21T01:32:02.362 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2024-06-21T01:32:02.397 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-06-21T01:32:02.492 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-21T01:32:02.544 INFO:teuthology.run_tasks:Running task kernel... 2024-06-21T01:32:02.558 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-21T01:32:02.558 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-21T01:32:02.558 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-21T01:32:02.559 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-21T01:32:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-21T01:32:02.575 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-06-21T01:32:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-21T01:32:02.576 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-06-21T01:32:02.632 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-437.el9.x86_64 2024-06-21T01:32:02.632 INFO:teuthology.task.kernel:Running kernel on smithi092: 5.14.0-437.el9.x86_64 2024-06-21T01:32:02.632 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum install -y kernel 2024-06-21T01:32:02.633 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-437.el9.x86_64 2024-06-21T01:32:02.633 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-437.el9.x86_64 2024-06-21T01:32:02.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-06-21T01:32:05.534 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.1 MB 00:01 2024-06-21T01:32:08.064 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.1 MB 00:03 2024-06-21T01:32:09.034 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 20 MB 00:02 2024-06-21T01:32:13.209 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.3 MB 00:00 2024-06-21T01:32:14.256 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 20 MB 00:05 2024-06-21T01:32:14.894 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 17 kB 00:00 2024-06-21T01:32:16.813 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-06-21T01:32:20.089 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.3 MB 00:02 2024-06-21T01:32:22.028 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-06-21T01:32:22.183 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 17 kB 00:00 2024-06-21T01:32:24.076 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-21T01:32:24.076 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-21T01:32:24.175 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-06-21T01:32:24.179 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-06-21T01:32:24.179 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2024-06-21T01:32:24.179 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-06-21T01:32:24.180 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-06-21T01:32:24.180 INFO:teuthology.orchestra.run.smithi092.stdout: kernel x86_64 5.14.0-460.el9 baseos 1.1 M 2024-06-21T01:32:24.180 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-06-21T01:32:24.180 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core x86_64 5.14.0-460.el9 baseos 16 M 2024-06-21T01:32:24.180 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules x86_64 5.14.0-460.el9 baseos 35 M 2024-06-21T01:32:24.180 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core x86_64 5.14.0-460.el9 baseos 29 M 2024-06-21T01:32:24.180 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:32:24.180 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-06-21T01:32:24.180 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-06-21T01:32:24.180 INFO:teuthology.orchestra.run.smithi092.stdout:Install 4 Packages 2024-06-21T01:32:24.180 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:32:24.181 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 81 M 2024-06-21T01:32:24.181 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 124 M 2024-06-21T01:32:24.181 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-06-21T01:32:24.730 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 21 MB 00:02 2024-06-21T01:32:25.279 INFO:teuthology.orchestra.run.smithi092.stdout:(1/4): kernel-5.14.0-460.el9.x86_64.rpm 1.1 MB/s | 1.1 MB 00:00 2024-06-21T01:32:25.946 INFO:teuthology.orchestra.run.smithi092.stdout:(2/4): kernel-core-5.14.0-460.el9.x86_64.rpm 9.8 MB/s | 16 MB 00:01 2024-06-21T01:32:26.796 INFO:teuthology.orchestra.run.smithi092.stdout:(3/4): kernel-modules-5.14.0-460.el9.x86_64.rpm 14 MB/s | 35 MB 00:02 2024-06-21T01:32:28.713 INFO:teuthology.orchestra.run.smithi092.stdout:(4/4): kernel-modules-core-5.14.0-460.el9.x86_6 8.5 MB/s | 29 MB 00:03 2024-06-21T01:32:28.713 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-06-21T01:32:28.713 INFO:teuthology.orchestra.run.smithi092.stdout:Total 18 MB/s | 81 MB 00:04 2024-06-21T01:32:29.422 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-06-21T01:32:29.783 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-06-21T01:32:29.783 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-06-21T01:32:30.133 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 6.6 kB/s | 1.7 kB 00:00 2024-06-21T01:32:30.723 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-06-21T01:32:30.724 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-06-21T01:32:32.164 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-21T01:32:32.164 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-21T01:32:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-21T01:32:32.266 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-21T01:32:32.266 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-06-21T01:32:32.266 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-21T01:32:32.266 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-21T01:32:32.266 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-460.el9 baseos 1.1 M 2024-06-21T01:32:32.266 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-06-21T01:32:32.266 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 5.14.0-460.el9 baseos 16 M 2024-06-21T01:32:32.266 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 5.14.0-460.el9 baseos 35 M 2024-06-21T01:32:32.266 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core x86_64 5.14.0-460.el9 baseos 29 M 2024-06-21T01:32:32.267 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-21T01:32:32.267 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-21T01:32:32.267 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-21T01:32:32.267 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-06-21T01:32:32.267 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-21T01:32:32.267 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 81 M 2024-06-21T01:32:32.267 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 124 M 2024-06-21T01:32:32.267 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-21T01:32:33.135 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-06-21T01:32:33.351 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): kernel-5.14.0-460.el9.x86_64.rpm 1.5 MB/s | 1.1 MB 00:00 2024-06-21T01:32:33.826 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-core-5.14.0-460.el9.x86_64 1/4 2024-06-21T01:32:33.921 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-21T01:32:34.043 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): kernel-core-5.14.0-460.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-06-21T01:32:34.685 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): kernel-modules-core-5.14.0-460.el9.x86_6 22 MB/s | 29 MB 00:01 2024-06-21T01:32:35.143 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): kernel-modules-5.14.0-460.el9.x86_64.rpm 14 MB/s | 35 MB 00:02 2024-06-21T01:32:35.144 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-21T01:32:35.144 INFO:teuthology.orchestra.run.smithi173.stdout:Total 28 MB/s | 81 MB 00:02 2024-06-21T01:32:35.853 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-21T01:32:36.194 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-21T01:32:36.194 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-21T01:32:36.628 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-21T01:32:36.773 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-21T01:32:37.167 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-21T01:32:37.168 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-21T01:32:39.567 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-21T01:32:40.180 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-core-5.14.0-460.el9.x86_64 1/4 2024-06-21T01:32:40.277 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-21T01:32:41.489 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-21T01:32:41.545 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:32:42.938 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-21T01:32:43.078 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-21T01:32:46.307 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:32:47.813 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-21T01:32:47.860 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:32:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:33:15.533 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:33:15.572 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:33:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:33:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-5.14.0-460.el9.x86_64 1/4 2024-06-21T01:33:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-21T01:33:15.904 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-21T01:33:16.743 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:33:16.743 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:33:16.743 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-06-21T01:33:16.743 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-5.14.0-460.el9.x86_64 2024-06-21T01:33:16.743 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core-5.14.0-460.el9.x86_64 2024-06-21T01:33:16.743 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-5.14.0-460.el9.x86_64 2024-06-21T01:33:16.743 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core-5.14.0-460.el9.x86_64 2024-06-21T01:33:16.743 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:33:16.744 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-06-21T01:33:17.115 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-06-21T01:33:17.190 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-21T01:33:17.190 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-21T01:33:17.191 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-460.el9.x86_64 2024-06-21T01:33:17.191 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-21T01:33:17.191 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-21T01:33:17.191 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-21T01:33:17.191 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2024-06-21T01:33:17.191 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-06-21T01:33:17.246 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-21T01:33:17.247 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-21T01:33:17.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -qi grub2-tools 2024-06-21T01:33:17.288 INFO:teuthology.orchestra.run.smithi092.stdout:Name : grub2-tools 2024-06-21T01:33:17.289 INFO:teuthology.orchestra.run.smithi092.stdout:Epoch : 1 2024-06-21T01:33:17.289 INFO:teuthology.orchestra.run.smithi092.stdout:Version : 2.06 2024-06-21T01:33:17.289 INFO:teuthology.orchestra.run.smithi092.stdout:Release : 68.el9 2024-06-21T01:33:17.289 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: x86_64 2024-06-21T01:33:17.289 INFO:teuthology.orchestra.run.smithi092.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-21T01:33:17.289 INFO:teuthology.orchestra.run.smithi092.stdout:Group : Unspecified 2024-06-21T01:33:17.290 INFO:teuthology.orchestra.run.smithi092.stdout:Size : 8264254 2024-06-21T01:33:17.290 INFO:teuthology.orchestra.run.smithi092.stdout:License : GPLv3+ 2024-06-21T01:33:17.290 INFO:teuthology.orchestra.run.smithi092.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-21T01:33:17.290 INFO:teuthology.orchestra.run.smithi092.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-21T01:33:17.290 INFO:teuthology.orchestra.run.smithi092.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-21T01:33:17.293 INFO:teuthology.orchestra.run.smithi092.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-21T01:33:17.294 INFO:teuthology.orchestra.run.smithi092.stdout:Packager : builder@centos.org 2024-06-21T01:33:17.294 INFO:teuthology.orchestra.run.smithi092.stdout:Vendor : CentOS 2024-06-21T01:33:17.294 INFO:teuthology.orchestra.run.smithi092.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-21T01:33:17.294 INFO:teuthology.orchestra.run.smithi092.stdout:Summary : Support tools for GRUB. 2024-06-21T01:33:17.294 INFO:teuthology.orchestra.run.smithi092.stdout:Description : 2024-06-21T01:33:17.294 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:33:17.295 INFO:teuthology.orchestra.run.smithi092.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-21T01:33:17.295 INFO:teuthology.orchestra.run.smithi092.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-21T01:33:17.295 INFO:teuthology.orchestra.run.smithi092.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-21T01:33:17.296 INFO:teuthology.orchestra.run.smithi092.stdout:hardware devices. 2024-06-21T01:33:17.296 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:33:17.296 INFO:teuthology.orchestra.run.smithi092.stdout:This subpackage provides tools for support of all platforms. 2024-06-21T01:33:17.297 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-21T01:33:17.298 INFO:teuthology.task.kernel:Updating grub on smithi092 to boot 5.14.0-460.el9.x86_64 2024-06-21T01:33:17.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-21T01:33:17.891 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2024-06-21T01:33:19.204 INFO:teuthology.orchestra.run.smithi092.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-21T01:33:19.234 INFO:teuthology.orchestra.run.smithi092.stderr:done 2024-06-21T01:33:19.237 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-06-21T01:33:19.252 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.wvag22Om95 2024-06-21T01:33:19.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wvag22Om95 2024-06-21T01:33:19.317 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.wvag22Om95 2024-06-21T01:33:19.463 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.wvag22Om95 is 6KB 2024-06-21T01:33:19.511 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.wvag22Om95 2024-06-21T01:33:19.525 DEBUG:teuthology.orchestra.run.smithi092:> sudo /bin/ls /boot/loader/entries || true 2024-06-21T01:33:19.586 INFO:teuthology.orchestra.run.smithi092.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-21T01:33:19.586 INFO:teuthology.orchestra.run.smithi092.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-21T01:33:19.586 INFO:teuthology.orchestra.run.smithi092.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-21T01:33:19.586 INFO:teuthology.orchestra.run.smithi092.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64.conf 2024-06-21T01:33:19.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64 2024-06-21T01:33:19.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2024-06-21T01:33:22.043 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:33:22.083 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:33:22.381 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:33:22.381 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-460.el9.x86_64 1/4 2024-06-21T01:33:22.382 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-21T01:33:22.382 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-21T01:33:23.188 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-21T01:33:23.189 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-21T01:33:23.189 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-21T01:33:23.189 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-460.el9.x86_64 2024-06-21T01:33:23.189 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core-5.14.0-460.el9.x86_64 2024-06-21T01:33:23.189 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-5.14.0-460.el9.x86_64 2024-06-21T01:33:23.189 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core-5.14.0-460.el9.x86_64 2024-06-21T01:33:23.189 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-21T01:33:23.189 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-21T01:33:23.563 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-06-21T01:33:23.662 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-21T01:33:23.662 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-21T01:33:23.663 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-460.el9.x86_64 2024-06-21T01:33:23.663 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-21T01:33:23.663 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-21T01:33:23.663 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-06-21T01:33:23.663 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-06-21T01:33:23.663 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-06-21T01:33:23.717 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-21T01:33:23.717 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-21T01:33:23.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-06-21T01:33:23.747 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-06-21T01:33:23.747 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-06-21T01:33:23.747 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-06-21T01:33:23.747 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 68.el9 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8264254 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-06-21T01:33:23.748 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-06-21T01:33:23.749 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-21T01:33:23.749 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-21T01:33:23.749 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-21T01:33:23.749 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-21T01:33:23.749 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-06-21T01:33:23.749 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-21T01:33:23.749 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-06-21T01:33:23.750 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-21T01:33:23.750 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 5.14.0-460.el9.x86_64 2024-06-21T01:33:23.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-21T01:33:24.332 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-06-21T01:33:25.705 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-21T01:33:25.731 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-06-21T01:33:25.733 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-06-21T01:33:25.764 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.YsHf5Dsf34 2024-06-21T01:33:25.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YsHf5Dsf34 2024-06-21T01:33:25.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.YsHf5Dsf34 2024-06-21T01:33:25.977 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.YsHf5Dsf34 is 6KB 2024-06-21T01:33:26.026 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.YsHf5Dsf34 2024-06-21T01:33:26.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-06-21T01:33:26.106 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-21T01:33:26.106 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-21T01:33:26.106 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-21T01:33:26.106 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64.conf 2024-06-21T01:33:26.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64 2024-06-21T01:33:26.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-06-21T01:33:49.753 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-21T01:33:49.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-06-21T01:33:49.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-21T01:33:56.266 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-06-21T01:33:56.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-21T01:33:56.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-21T01:34:08.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-06-21T01:34:14.819 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-06-21T01:34:17.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-06-21T01:34:17.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-21T01:34:20.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-06-21T01:34:23.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-21T01:34:23.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-21T01:34:26.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-06-21T01:34:32.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-06-21T01:34:32.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-21T01:34:38.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-21T01:34:38.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-21T01:34:57.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-06-21T01:35:04.480 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-06-21T01:35:04.974 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-06-21T01:35:04.974 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-460.el9.x86_64"... 2024-06-21T01:35:04.975 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-06-21T01:35:05.007 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-460.el9.x86_64 2024-06-21T01:35:05.007 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-460.el9.x86_64 vs 5.14.0-460.el9.x86_64 2024-06-21T01:35:05.007 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-21T01:35:05.008 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-21T01:35:06.008 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-21T01:35:06.008 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-21T01:35:06.062 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2024-06-21T01:35:06.123 DEBUG:teuthology.parallel:result is None 2024-06-21T01:35:12.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-21T01:35:12.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-21T01:35:12.665 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-06-21T01:35:13.047 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-21T01:35:13.047 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-460.el9.x86_64"... 2024-06-21T01:35:13.048 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-06-21T01:35:13.062 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-460.el9.x86_64 2024-06-21T01:35:13.063 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-460.el9.x86_64 vs 5.14.0-460.el9.x86_64 2024-06-21T01:35:13.063 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-21T01:35:13.063 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-21T01:35:14.064 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-06-21T01:35:14.064 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-21T01:35:14.168 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-06-21T01:35:14.241 DEBUG:teuthology.parallel:result is None 2024-06-21T01:35:14.241 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-21T01:35:14.249 INFO:teuthology.task.internal:Creating test directory... 2024-06-21T01:35:14.249 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-21T01:35:14.252 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-21T01:35:14.268 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-21T01:35:14.293 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-21T01:35:14.325 INFO:teuthology.task.internal:Creating archive directory... 2024-06-21T01:35:14.325 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-21T01:35:14.328 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-21T01:35:14.357 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-21T01:35:14.363 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-21T01:35:14.363 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-21T01:35:14.393 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-21T01:35:14.430 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-21T01:35:14.442 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-21T01:35:14.532 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-21T01:35:14.542 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-21T01:35:14.543 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-21T01:35:14.550 INFO:teuthology.task.internal:Configuring sudo... 2024-06-21T01:35:14.550 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-21T01:35:14.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-21T01:35:14.579 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-21T01:35:14.588 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-21T01:35:14.588 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-21T01:35:14.621 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-21T01:35:14.637 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-21T01:35:14.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-21T01:35:14.775 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-21T01:35:14.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-21T01:35:14.901 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-06-21T01:35:14.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-21T01:35:14.967 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-21T01:35:15.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-21T01:35:15.056 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-21T01:35:15.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-21T01:35:15.185 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-21T01:35:15.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-21T01:35:15.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2024-06-21T01:35:15.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-06-21T01:35:15.317 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-21T01:35:15.342 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-21T01:35:15.634 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-21T01:35:15.642 INFO:teuthology.task.internal:Starting timer... 2024-06-21T01:35:15.642 INFO:teuthology.run_tasks:Running task pcp... 2024-06-21T01:35:15.689 INFO:teuthology.run_tasks:Running task selinux... 2024-06-21T01:35:15.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2024-06-21T01:35:15.818 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2024-06-21T01:35:15.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-06-21T01:35:16.001 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-06-21T01:35:16.003 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-21T01:35:16.003 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2024-06-21T01:35:16.027 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2024-06-21T01:35:16.027 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-06-21T01:35:16.078 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-06-21T01:35:16.079 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi092.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-06-21T01:35:16.079 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-21T01:35:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-21T01:35:16.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-21T01:35:16.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-21T01:35:16.135 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-21T01:35:16.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2024-06-21T01:35:16.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-06-21T01:35:16.418 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-21T01:35:16.427 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-21T01:35:16.429 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-21T01:35:16.538 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-21T01:35:16.549 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-21T01:35:16.550 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi092.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-21T01:39:43.940 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-06-21T01:39:43.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-06-21T01:39:43.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-21T01:39:44.023 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-06-21T01:39:44.102 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-06-21T01:39:44.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-21T01:39:44.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-21T01:39:44.176 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-06-21T01:39:44.259 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-21T01:39:44.260 INFO:teuthology.run_tasks:Running task clock... 2024-06-21T01:39:44.269 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-21T01:39:44.270 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-21T01:39:44.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-21T01:39:44.273 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-21T01:39:44.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-21T01:39:44.299 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-21T01:39:44.313 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-21T01:39:44.330 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-21T01:39:44.337 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2024-06-21T01:39:44.346 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-21T01:39:44.348 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2024-06-21T01:39:44.363 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-21T01:39:44.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-06-21T01:39:44.376 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-21T01:39:44.381 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-06-21T01:39:44.396 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-21T01:39:44.410 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-21T01:39:44.436 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-06-21T01:39:44.439 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-21T01:39:44.439 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-06-21T01:39:44.439 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-21T01:39:44.439 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-21T01:39:44.439 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-21T01:39:44.439 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-21T01:39:44.472 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-06-21T01:39:44.475 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-21T01:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-06-21T01:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-21T01:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-21T01:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-21T01:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-21T01:39:44.477 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-21T01:39:44.573 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': 'c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd'} 2024-06-21T01:39:44.573 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-06-21T01:39:44.573 INFO:tasks.cephadm:Cluster fsid is 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:39:44.573 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-21T01:39:44.574 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.92', 'mon.c': '[v2:172.21.15.92:3301,v1:172.21.15.92:6790]', 'mon.b': '172.21.15.173'} 2024-06-21T01:39:44.574 INFO:tasks.cephadm:First mon is mon.a on smithi092 2024-06-21T01:39:44.574 INFO:tasks.cephadm:First mgr is y 2024-06-21T01:39:44.574 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-21T01:39:44.574 DEBUG:teuthology.orchestra.run.smithi092:> sudo hostname $(hostname -s) 2024-06-21T01:39:44.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-06-21T01:39:44.626 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-06-21T01:39:44.626 DEBUG:teuthology.orchestra.run.smithi092:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-21T01:39:45.159 INFO:teuthology.orchestra.run.smithi092.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Jun 21 01:39 /home/ubuntu/cephtest/cephadm 2024-06-21T01:39:45.159 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-21T01:39:45.457 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Jun 21 01:39 /home/ubuntu/cephtest/cephadm 2024-06-21T01:39:45.457 DEBUG:teuthology.orchestra.run.smithi092:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-21T01:39:45.475 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-21T01:39:45.502 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-06-21T01:39:45.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-06-21T01:39:45.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-06-21T01:39:46.410 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-06-21T01:39:46.425 INFO:teuthology.orchestra.run.smithi092.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-06-21T01:40:02.945 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:40:02.945 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-06-21T01:40:02.945 INFO:teuthology.orchestra.run.smithi092.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-06-21T01:40:02.945 INFO:teuthology.orchestra.run.smithi092.stdout: "repo_digests": [ 2024-06-21T01:40:02.945 INFO:teuthology.orchestra.run.smithi092.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-06-21T01:40:02.945 INFO:teuthology.orchestra.run.smithi092.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-06-21T01:40:02.945 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-06-21T01:40:02.946 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:40:07.420 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-06-21T01:40:07.420 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-06-21T01:40:07.420 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-06-21T01:40:07.420 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-06-21T01:40:07.420 INFO:teuthology.orchestra.run.smithi173.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-06-21T01:40:07.420 INFO:teuthology.orchestra.run.smithi173.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-06-21T01:40:07.420 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-06-21T01:40:07.421 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-06-21T01:40:07.440 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph 2024-06-21T01:40:07.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-06-21T01:40:07.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /etc/ceph 2024-06-21T01:40:07.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-06-21T01:40:07.566 INFO:tasks.cephadm:Writing seed config... 2024-06-21T01:40:07.568 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-06-21T01:40:07.568 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-21T01:40:07.569 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-21T01:40:07.569 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-06-21T01:40:07.569 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-21T01:40:07.569 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-21T01:40:07.569 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-21T01:40:07.569 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-21T01:40:07.569 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-21T01:40:07.570 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-06-21T01:40:07.570 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-21T01:40:07.598 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 22eef780-2f6f-11ef-bc9d-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-06-21T01:40:07.598 DEBUG:teuthology.orchestra.run.smithi092:mon.a> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.a.service 2024-06-21T01:40:07.640 DEBUG:teuthology.orchestra.run.smithi092:mgr.y> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.y.service 2024-06-21T01:40:07.683 INFO:tasks.cephadm:Bootstrapping... 2024-06-21T01:40:07.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.92 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-21T01:40:07.876 INFO:teuthology.orchestra.run.smithi092.stderr:-------------------------------------------------------------------------------- 2024-06-21T01:40:07.877 INFO:teuthology.orchestra.run.smithi092.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '22eef780-2f6f-11ef-bc9d-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.92', '--skip-admin-label'] 2024-06-21T01:40:07.900 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: 5.1.0 2024-06-21T01:40:07.902 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying podman|docker is present... 2024-06-21T01:40:07.924 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: 5.1.0 2024-06-21T01:40:07.927 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying lvm2 is present... 2024-06-21T01:40:07.927 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying time synchronization is in place... 2024-06-21T01:40:07.934 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-06-21T01:40:07.940 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-06-21T01:40:07.947 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: enabled 2024-06-21T01:40:07.953 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: active 2024-06-21T01:40:07.953 INFO:teuthology.orchestra.run.smithi092.stderr:Unit chronyd.service is enabled and running 2024-06-21T01:40:07.954 INFO:teuthology.orchestra.run.smithi092.stderr:Repeating the final host check... 2024-06-21T01:40:07.975 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: 5.1.0 2024-06-21T01:40:07.977 INFO:teuthology.orchestra.run.smithi092.stderr:podman (/bin/podman) version 5.1.0 is present 2024-06-21T01:40:07.977 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl is present 2024-06-21T01:40:07.977 INFO:teuthology.orchestra.run.smithi092.stderr:lvcreate is present 2024-06-21T01:40:07.983 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-06-21T01:40:07.989 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-06-21T01:40:07.996 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: enabled 2024-06-21T01:40:08.002 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: active 2024-06-21T01:40:08.003 INFO:teuthology.orchestra.run.smithi092.stderr:Unit chronyd.service is enabled and running 2024-06-21T01:40:08.003 INFO:teuthology.orchestra.run.smithi092.stderr:Host looks OK 2024-06-21T01:40:08.003 INFO:teuthology.orchestra.run.smithi092.stderr:Cluster fsid: 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:40:08.003 INFO:teuthology.orchestra.run.smithi092.stderr:Acquiring lock 140406972228608 on /run/cephadm/22eef780-2f6f-11ef-bc9d-c7b262605968.lock 2024-06-21T01:40:08.003 INFO:teuthology.orchestra.run.smithi092.stderr:Lock 140406972228608 acquired on /run/cephadm/22eef780-2f6f-11ef-bc9d-c7b262605968.lock 2024-06-21T01:40:08.004 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying IP 172.21.15.92 port 3300 ... 2024-06-21T01:40:08.004 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying IP 172.21.15.92 port 6789 ... 2024-06-21T01:40:08.004 INFO:teuthology.orchestra.run.smithi092.stderr:Base mon IP is 172.21.15.92, final addrv is [v2:172.21.15.92:3300,v1:172.21.15.92:6789] 2024-06-21T01:40:08.006 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.92 metric 100 2024-06-21T01:40:08.007 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.92 metric 100 2024-06-21T01:40:08.009 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-06-21T01:40:08.009 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-21T01:40:08.009 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium 2024-06-21T01:40:08.012 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-21T01:40:08.012 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-06-21T01:40:08.012 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-06-21T01:40:08.012 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-21T01:40:08.012 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:cd45/64 scope link 2024-06-21T01:40:08.012 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-06-21T01:40:08.013 INFO:teuthology.orchestra.run.smithi092.stderr:Mon IP `172.21.15.92` is in CIDR network `172.21.0.0/20` 2024-06-21T01:40:08.013 INFO:teuthology.orchestra.run.smithi092.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-21T01:40:08.013 INFO:teuthology.orchestra.run.smithi092.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-06-21T01:40:08.116 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-06-21T01:40:08.707 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Getting image source signatures 2024-06-21T01:40:08.707 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-06-21T01:40:08.707 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-06-21T01:40:08.707 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-06-21T01:40:08.708 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-06-21T01:40:08.708 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-06-21T01:40:08.708 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-06-21T01:40:08.863 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Writing manifest to image destination 2024-06-21T01:40:09.015 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-06-21T01:40:09.413 INFO:teuthology.orchestra.run.smithi092.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-06-21T01:40:09.906 INFO:teuthology.orchestra.run.smithi092.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-06-21T01:40:09.906 INFO:teuthology.orchestra.run.smithi092.stderr:Extracting ceph user uid/gid from container image... 2024-06-21T01:40:10.298 INFO:teuthology.orchestra.run.smithi092.stderr:stat: 167 167 2024-06-21T01:40:10.887 INFO:teuthology.orchestra.run.smithi092.stderr:Creating initial keys... 2024-06-21T01:40:11.261 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph-authtool: AQD72XRmDgq7ChAAMq34Pl5Hgkv0Cobp2Ak/9Q== 2024-06-21T01:40:12.233 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph-authtool: AQD82XRmdEc8CRAA2wdX3xx5vqt87nzD5AlcDg== 2024-06-21T01:40:13.131 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph-authtool: AQD92XRmCX/8AhAAGgydlZ2bgu3L4wOI2VW4xw== 2024-06-21T01:40:13.621 INFO:teuthology.orchestra.run.smithi092.stderr:Creating initial monmap... 2024-06-21T01:40:14.003 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-21T01:40:14.004 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-06-21T01:40:14.004 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:40:14.004 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-21T01:40:14.485 INFO:teuthology.orchestra.run.smithi092.stderr:monmaptool for a [v2:172.21.15.92:3300,v1:172.21.15.92:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-21T01:40:14.485 INFO:teuthology.orchestra.run.smithi092.stderr:setting min_mon_release = octopus 2024-06-21T01:40:14.485 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: set fsid to 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:40:14.485 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-21T01:40:14.485 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-06-21T01:40:14.485 INFO:teuthology.orchestra.run.smithi092.stderr:Creating mon... 2024-06-21T01:40:15.493 INFO:teuthology.orchestra.run.smithi092.stderr:create mon.a on 2024-06-21T01:40:15.715 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-21T01:40:15.868 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968.target. 2024-06-21T01:40:15.868 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968.target. 2024-06-21T01:40:16.175 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Failed to reset failed state of unit ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.a.service: Unit ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.a.service not loaded. 2024-06-21T01:40:16.183 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Created symlink /etc/systemd/system/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968.target.wants/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.a.service → /etc/systemd/system/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@.service. 2024-06-21T01:40:16.898 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: disabled 2024-06-21T01:40:16.905 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-06-21T01:40:16.906 INFO:teuthology.orchestra.run.smithi092.stderr:firewalld.service is not enabled 2024-06-21T01:40:16.906 INFO:teuthology.orchestra.run.smithi092.stderr:Not possible to enable service . firewalld.service is not available 2024-06-21T01:40:16.907 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mon to start... 2024-06-21T01:40:16.907 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mon... 2024-06-21T01:40:16.950 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:16 smithi092 systemd[1]: Started Ceph mon.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:40:17.400 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: cluster: 2024-06-21T01:40:17.400 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: id: 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:40:17.400 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-06-21T01:40:17.400 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:17.400 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: services: 2024-06-21T01:40:17.400 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.572951s) 2024-06-21T01:40:17.400 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mgr: no daemons active 2024-06-21T01:40:17.400 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-06-21T01:40:17.401 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:17.401 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: data: 2024-06-21T01:40:17.401 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-06-21T01:40:17.401 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-06-21T01:40:17.401 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-06-21T01:40:17.401 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: pgs: 2024-06-21T01:40:17.401 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:17.934 INFO:teuthology.orchestra.run.smithi092.stderr:mon is available 2024-06-21T01:40:17.934 INFO:teuthology.orchestra.run.smithi092.stderr:Assimilating anything we can from ceph.conf... 2024-06-21T01:40:18.057 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:17 smithi092 ceph-mon[24763]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-21T01:40:18.058 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:17 smithi092 ceph-mon[24763]: monmap e1: 1 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0]} 2024-06-21T01:40:18.058 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:17 smithi092 ceph-mon[24763]: fsmap 2024-06-21T01:40:18.058 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:17 smithi092 ceph-mon[24763]: osdmap e1: 0 total, 0 up, 0 in 2024-06-21T01:40:18.058 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:17 smithi092 ceph-mon[24763]: mgrmap e1: no daemons active 2024-06-21T01:40:18.058 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:17 smithi092 ceph-mon[24763]: from='client.? 172.21.15.92:0/1015510131' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-21T01:40:18.448 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:18.448 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [global] 2024-06-21T01:40:18.448 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: fsid = 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:40:18.448 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.92:3300,v1:172.21.15.92:6789] 2024-06-21T01:40:18.448 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-06-21T01:40:18.448 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-06-21T01:40:18.448 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-06-21T01:40:18.449 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-06-21T01:40:18.449 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:18.449 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [mgr] 2024-06-21T01:40:18.449 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-06-21T01:40:18.449 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-06-21T01:40:18.449 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:18.449 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [osd] 2024-06-21T01:40:18.449 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-06-21T01:40:18.449 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-06-21T01:40:19.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:18 smithi092 ceph-mon[24763]: from='client.? 172.21.15.92:0/511703575' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-21T01:40:19.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:18 smithi092 ceph-mon[24763]: from='client.? 172.21.15.92:0/511703575' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-21T01:40:19.006 INFO:teuthology.orchestra.run.smithi092.stderr:Generating new minimal ceph.conf... 2024-06-21T01:40:20.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:19 smithi092 ceph-mon[24763]: from='client.? 172.21.15.92:0/743036204' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:40:20.104 INFO:teuthology.orchestra.run.smithi092.stderr:Restarting the monitor... 2024-06-21T01:40:20.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 systemd[1]: Stopping Ceph mon.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:40:20.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 bash[24983]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon.a" found: no such container 2024-06-21T01:40:20.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a[24759]: 2024-06-21T01:40:20.266+0000 7efeff262700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:40:20.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a[24759]: 2024-06-21T01:40:20.266+0000 7efeff262700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-21T01:40:20.679 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 podman[24996]: 2024-06-21 01:40:20.35873918 +0000 UTC m=+0.196100094 container died 5fffaf7941f0785df99bf962be89dc4443ca7d7f719fa5d0e1f4e5c27c162820 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.buildah.version=1.19.8, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, release=754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-06-21T01:40:20.679 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 podman[24996]: 2024-06-21 01:40:20.682956977 +0000 UTC m=+0.520317926 container cleanup 5fffaf7941f0785df99bf962be89dc4443ca7d7f719fa5d0e1f4e5c27c162820 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.buildah.version=1.19.8, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, distribution-scope=public, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, architecture=x86_64) 2024-06-21T01:40:21.097 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 bash[24996]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a 2024-06-21T01:40:21.097 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 bash[25023]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon.a" found: no such container 2024-06-21T01:40:21.098 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 podman[25008]: 2024-06-21 01:40:20.858709538 +0000 UTC m=+0.500455510 container remove 5fffaf7941f0785df99bf962be89dc4443ca7d7f719fa5d0e1f4e5c27c162820 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-type=git, release=754, GIT_CLEAN=True, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, RELEASE=HEAD, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870) 2024-06-21T01:40:21.098 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-21T01:40:21.098 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 systemd[1]: Stopped Ceph mon.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:40:21.098 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:20 smithi092 systemd[1]: Starting Ceph mon.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:40:21.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 podman[25089]: 2024-06-21 01:40:21.101182592 +0000 UTC m=+0.113825940 container create 3daa4a891e57574c0c016dd9c8883dc298b55abb936cbcf1163ec8973d8ae928 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, release=754, com.redhat.component=centos-stream-container, GIT_CLEAN=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, ceph=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, version=8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux ) 2024-06-21T01:40:21.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 podman[25089]: 2024-06-21 01:40:21.00502569 +0000 UTC m=+0.017669009 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-06-21T01:40:21.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 podman[25089]: 2024-06-21 01:40:21.319954985 +0000 UTC m=+0.332598300 container init 3daa4a891e57574c0c016dd9c8883dc298b55abb936cbcf1163ec8973d8ae928 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, distribution-scope=public, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , architecture=x86_64, io.k8s.display-name=CentOS Stream 8) 2024-06-21T01:40:21.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 podman[25089]: 2024-06-21 01:40:21.323030088 +0000 UTC m=+0.335673391 container start 3daa4a891e57574c0c016dd9c8883dc298b55abb936cbcf1163ec8973d8ae928 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.buildah.version=1.19.8, version=8, RELEASE=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8) 2024-06-21T01:40:21.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: set uid:gid to 167:167 (ceph:ceph) 2024-06-21T01:40:21.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-06-21T01:40:21.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: pidfile_write: ignore empty --pid-file 2024-06-21T01:40:21.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: load: jerasure load: lrc 2024-06-21T01:40:21.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: RocksDB version: 6.15.5 2024-06-21T01:40:21.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Compile date Apr 18 2022 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: DB SUMMARY 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: DB Session ID: 9P8KAZLH679GG5ENR9PW 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: CURRENT file: CURRENT 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: IDENTITY file: IDENTITY 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82623 ; 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.error_if_exists: 0 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.create_if_missing: 0 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.paranoid_checks: 1 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-21T01:40:21.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.env: 0x560ee16c6860 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.fs: Posix File System 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.info_log: 0x560ee37bfdc0 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_file_opening_threads: 16 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.statistics: (nil) 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.use_fsync: 0 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_log_file_size: 0 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.keep_log_file_num: 1000 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.recycle_log_file_num: 0 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.allow_fallocate: 1 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.allow_mmap_reads: 0 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.allow_mmap_writes: 0 2024-06-21T01:40:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.use_direct_reads: 0 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.create_missing_column_families: 0 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.db_log_dir: 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.advise_random_on_open: 1 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.db_write_buffer_size: 0 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.write_buffer_manager: 0x560ee38b02a0 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-21T01:40:21.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.rate_limiter: (nil) 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.wal_recovery_mode: 2 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.enable_thread_tracking: 0 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.enable_pipelined_write: 0 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.unordered_write: 0 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.row_cache: None 2024-06-21T01:40:21.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.wal_filter: None 2024-06-21T01:40:21.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-21T01:40:21.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.allow_ingest_behind: 0 2024-06-21T01:40:21.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.preserve_deletes: 0 2024-06-21T01:40:21.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.two_write_queues: 0 2024-06-21T01:40:21.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.manual_wal_flush: 0 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.atomic_flush: 0 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.log_readahead_size: 0 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.best_efforts_recovery: 0 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.allow_data_in_errors: 0 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.db_host_id: __hostname__ 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_background_jobs: 2 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_background_compactions: -1 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_subcompactions: 1 2024-06-21T01:40:21.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-21T01:40:21.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-21T01:40:21.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-21T01:40:21.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_total_wal_size: 0 2024-06-21T01:40:21.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-21T01:40:21.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-21T01:40:21.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-21T01:40:21.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-21T01:40:21.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_open_files: -1 2024-06-21T01:40:21.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.bytes_per_sync: 0 2024-06-21T01:40:21.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-21T01:40:21.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_readahead_size: 0 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_background_flushes: -1 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Compression algorithms supported: 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: kZSTD supported: 0 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: kXpressCompression supported: 0 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: kLZ4HCCompression supported: 1 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: kLZ4Compression supported: 1 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: kBZip2Compression supported: 0 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: kZlibCompression supported: 1 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: kSnappyCompression supported: 1 2024-06-21T01:40:21.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.merge_operator: 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_filter: None 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_filter_factory: None 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.sst_partitioner_factory: None 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560ee378bd00) 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout: cache_index_and_filter_blocks: 1 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout: pin_top_level_index_and_filter: 1 2024-06-21T01:40:21.359 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_type: 0 2024-06-21T01:40:21.360 INFO:journalctl@ceph.mon.a.smithi092.stdout: data_block_index_type: 0 2024-06-21T01:40:21.360 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_shortening: 1 2024-06-21T01:40:21.360 INFO:journalctl@ceph.mon.a.smithi092.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-21T01:40:21.360 INFO:journalctl@ceph.mon.a.smithi092.stdout: hash_index_allow_collision: 1 2024-06-21T01:40:21.360 INFO:journalctl@ceph.mon.a.smithi092.stdout: checksum: 1 2024-06-21T01:40:21.360 INFO:journalctl@ceph.mon.a.smithi092.stdout: no_block_cache: 0 2024-06-21T01:40:21.360 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache: 0x560ee37f6170 2024-06-21T01:40:21.360 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_name: BinnedLRUCache 2024-06-21T01:40:21.360 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_options: 2024-06-21T01:40:21.360 INFO:journalctl@ceph.mon.a.smithi092.stdout: capacity : 536870912 2024-06-21T01:40:21.360 INFO:journalctl@ceph.mon.a.smithi092.stdout: num_shard_bits : 4 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: strict_capacity_limit : 0 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: high_pri_pool_ratio: 0.000 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_compressed: (nil) 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: persistent_cache: (nil) 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_size: 4096 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_size_deviation: 10 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_restart_interval: 16 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_block_restart_interval: 1 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: metadata_block_size: 4096 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: partition_filters: 0 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: use_delta_encoding: 1 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: whole_key_filtering: 1 2024-06-21T01:40:21.361 INFO:journalctl@ceph.mon.a.smithi092.stdout: verify_compression: 0 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout: read_amp_bytes_per_bit: 0 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout: format_version: 4 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout: enable_index_compression: 1 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_align: 0 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.write_buffer_size: 33554432 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_write_buffer_number: 2 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compression: NoCompression 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.bottommost_compression: Disabled 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.prefix_extractor: nullptr 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.num_levels: 7 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-21T01:40:21.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-21T01:40:21.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-21T01:40:21.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-21T01:40:21.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-21T01:40:21.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-21T01:40:21.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-21T01:40:21.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-21T01:40:21.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-21T01:40:21.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-21T01:40:21.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compression_opts.level: 32767 2024-06-21T01:40:21.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compression_opts.strategy: 0 2024-06-21T01:40:21.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compression_opts.enabled: false 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.target_file_size_base: 67108864 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-21T01:40:21.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.arena_block_size: 4194304 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.disable_auto_compactions: 0 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-21T01:40:21.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.table_properties_collectors: 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.inplace_update_support: 0 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-21T01:40:21.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-21T01:40:21.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-21T01:40:21.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.bloom_locality: 0 2024-06-21T01:40:21.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.max_successive_merges: 0 2024-06-21T01:40:21.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-21T01:40:21.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.paranoid_file_checks: 0 2024-06-21T01:40:21.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.force_consistency_checks: 1 2024-06-21T01:40:21.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.report_bg_io_stats: 0 2024-06-21T01:40:21.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.ttl: 2592000 2024-06-21T01:40:21.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-21T01:40:21.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.enable_blob_files: false 2024-06-21T01:40:21.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.min_blob_size: 0 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.blob_file_size: 268435456 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934021348805, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934021349480, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79173, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77436, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71745, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1718934021, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "545ebc10-8111-4c62-890d-5766c47d9cdf", "db_session_id": "9P8KAZLH679GG5ENR9PW"}} 2024-06-21T01:40:21.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934021349789, "job": 1, "event": "recovery_finished"} 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x560ee37dca80 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: DB pointer 0x560ee37ec000 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** DB Stats ** 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-21T01:40:21.369 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.0 0.00 0.00 1 0.000 0 0 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.0 0.00 0.00 1 0.000 0 0 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.0 0.00 0.00 1 0.000 0 0 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 167.0 0.00 0.00 1 0.000 0 0 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-21T01:40:21.370 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative compaction: 0.00 GB write, 25.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval compaction: 0.00 GB write, 25.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: L0 2/0 79.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.0 0.00 0.00 1 0.000 0 0 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: Sum 2/0 79.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.0 0.00 0.00 1 0.000 0 0 2024-06-21T01:40:21.371 INFO:journalctl@ceph.mon.a.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 167.0 0.00 0.00 1 0.000 0 0 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative compaction: 0.00 GB write, 25.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-21T01:40:21.372 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: starting mon.a rank 0 at public addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] at bind addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: mon.a@-1(???) e1 preinit fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: mon.a@-1(???).mds e1 new map 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: mon.a@-1(???).mds e1 print_map 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout: e1 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout: legacy client fscid: -1 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout: No filesystems configured 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-21T01:40:21.373 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta expand map: {default=false} 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta from 'false' to 'false' 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta expanded map: {default=false} 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta expand map: {default=info} 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta from 'info' to 'info' 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta expanded map: {default=info} 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta expand map: {default=daemon} 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta from 'daemon' to 'daemon' 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta expanded map: {default=daemon} 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta expand map: {default=debug} 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta from 'debug' to 'debug' 2024-06-21T01:40:21.374 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: expand_channel_meta expanded map: {default=debug} 2024-06-21T01:40:21.422 INFO:teuthology.orchestra.run.smithi092.stderr:Setting mon public_network to 172.21.0.0/20 2024-06-21T01:40:21.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-21T01:40:21.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: monmap e1: 1 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0]} 2024-06-21T01:40:21.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: fsmap 2024-06-21T01:40:21.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: osdmap e1: 0 total, 0 up, 0 in 2024-06-21T01:40:21.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 ceph-mon[25107]: mgrmap e1: no daemons active 2024-06-21T01:40:21.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 bash[25089]: 3daa4a891e57574c0c016dd9c8883dc298b55abb936cbcf1163ec8973d8ae928 2024-06-21T01:40:21.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:21 smithi092 systemd[1]: Started Ceph mon.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:40:23.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:22 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2577094691' entity='client.admin' 2024-06-21T01:40:27.161 INFO:teuthology.orchestra.run.smithi092.stderr:Wrote config to /etc/ceph/ceph.conf 2024-06-21T01:40:27.161 INFO:teuthology.orchestra.run.smithi092.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-21T01:40:27.161 INFO:teuthology.orchestra.run.smithi092.stderr:Creating mgr... 2024-06-21T01:40:27.162 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying port 9283 ... 2024-06-21T01:40:27.523 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Failed to reset failed state of unit ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.y.service: Unit ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.y.service not loaded. 2024-06-21T01:40:27.561 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Created symlink /etc/systemd/system/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968.target.wants/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@.service. 2024-06-21T01:40:28.355 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:28 smithi092 podman[25321]: 2024-06-21 01:40:28.050504784 +0000 UTC m=+0.017381705 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-06-21T01:40:29.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:29 smithi092 podman[25321]: 2024-06-21 01:40:29.037647183 +0000 UTC m=+1.004524146 container create 0b61fab2e6e278ec4c04732ad076bc96f91e693541e72f31beae689d31e4ddde (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, version=8, ceph=True, architecture=x86_64, vcs-type=git, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, release=754, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870) 2024-06-21T01:40:31.206 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:30 smithi092 podman[25321]: 2024-06-21 01:40:30.928260548 +0000 UTC m=+2.895137470 container init 0b61fab2e6e278ec4c04732ad076bc96f91e693541e72f31beae689d31e4ddde (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_CLEAN=True, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , release=754, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, name=centos-stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8) 2024-06-21T01:40:31.206 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:30 smithi092 podman[25321]: 2024-06-21 01:40:30.932528486 +0000 UTC m=+2.899405406 container start 0b61fab2e6e278ec4c04732ad076bc96f91e693541e72f31beae689d31e4ddde (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y, name=centos-stream, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, io.buildah.version=1.19.8, release=754, ceph=True, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, architecture=x86_64) 2024-06-21T01:40:31.206 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:31 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:31.080+0000 7f2fbf0dc000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:40:31.206 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:31 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:31.209+0000 7f2fbf0dc000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:40:31.380 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: disabled 2024-06-21T01:40:31.387 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-06-21T01:40:31.388 INFO:teuthology.orchestra.run.smithi092.stderr:firewalld.service is not enabled 2024-06-21T01:40:31.388 INFO:teuthology.orchestra.run.smithi092.stderr:Not possible to enable service . firewalld.service is not available 2024-06-21T01:40:31.396 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: disabled 2024-06-21T01:40:31.403 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-06-21T01:40:31.404 INFO:teuthology.orchestra.run.smithi092.stderr:firewalld.service is not enabled 2024-06-21T01:40:31.404 INFO:teuthology.orchestra.run.smithi092.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-06-21T01:40:31.404 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mgr to start... 2024-06-21T01:40:31.404 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mgr... 2024-06-21T01:40:31.530 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:31 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:31.307+0000 7f2fbf0dc000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:40:31.530 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:31 smithi092 bash[25321]: 0b61fab2e6e278ec4c04732ad076bc96f91e693541e72f31beae689d31e4ddde 2024-06-21T01:40:31.530 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:31 smithi092 systemd[1]: Started Ceph mgr.y for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:40:31.786 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:31 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:31.624+0000 7f2fbf0dc000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:40:31.787 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:31 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:31.710+0000 7f2fbf0dc000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:40:31.970 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsid": "22eef780-2f6f-11ef-bc9d-c7b262605968", 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "health": { 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "checks": {}, 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mutes": [] 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum": [ 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 0 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-06-21T01:40:31.971 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_names": [ 2024-06-21T01:40:31.972 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "a" 2024-06-21T01:40:31.972 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-06-21T01:40:31.972 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-06-21T01:40:31.972 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "monmap": { 2024-06-21T01:40:31.972 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:31.973 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-06-21T01:40:31.973 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_mons": 1 2024-06-21T01:40:31.973 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:31.973 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osdmap": { 2024-06-21T01:40:31.973 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:31.973 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_osds": 0, 2024-06-21T01:40:31.973 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-06-21T01:40:31.974 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-06-21T01:40:31.974 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-06-21T01:40:31.974 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-06-21T01:40:31.974 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-06-21T01:40:31.974 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:31.974 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgmap": { 2024-06-21T01:40:31.974 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-06-21T01:40:31.974 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-06-21T01:40:31.974 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pools": 0, 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_objects": 0, 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_total": 0 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsmap": { 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "by_rank": [], 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "up:standby": 0 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mgrmap": { 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "available": false, 2024-06-21T01:40:31.975 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modules": [ 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "iostat", 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "nfs", 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "restful" 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "servicemap": { 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modified": "2024-06-21T01:40:16.829925+0000", 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "progress_events": {} 2024-06-21T01:40:31.976 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-06-21T01:40:32.550 INFO:teuthology.orchestra.run.smithi092.stderr:mgr not available, waiting (1/15)... 2024-06-21T01:40:32.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:32 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1456418757' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-21T01:40:33.120 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:32 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:32.870+0000 7f2fbf0dc000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:40:33.120 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:33.033+0000 7f2fbf0dc000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:40:33.120 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:33.114+0000 7f2fbf0dc000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:40:33.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:33.437+0000 7f2fbf0dc000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:40:33.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:33.533+0000 7f2fbf0dc000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:40:33.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:33.629+0000 7f2fbf0dc000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:40:34.280 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:33.876+0000 7f2fbf0dc000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:40:34.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:34.324+0000 7f2fbf0dc000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:40:34.976 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:34.806+0000 7f2fbf0dc000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:40:34.976 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:34.901+0000 7f2fbf0dc000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:40:35.158 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:35.158 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsid": "22eef780-2f6f-11ef-bc9d-c7b262605968", 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "health": { 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "checks": {}, 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mutes": [] 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum": [ 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 0 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_names": [ 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "a" 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-06-21T01:40:35.159 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_age": 13, 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "monmap": { 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_mons": 1 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osdmap": { 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_osds": 0, 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-06-21T01:40:35.160 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgmap": { 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pools": 0, 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_objects": 0, 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_total": 0 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsmap": { 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "by_rank": [], 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "up:standby": 0 2024-06-21T01:40:35.161 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mgrmap": { 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "available": false, 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modules": [ 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "iostat", 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "nfs", 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "restful" 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "servicemap": { 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modified": "2024-06-21T01:40:16.829925+0000", 2024-06-21T01:40:35.162 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-06-21T01:40:35.163 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:35.163 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "progress_events": {} 2024-06-21T01:40:35.163 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-06-21T01:40:35.291 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:35 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/726438226' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-21T01:40:35.292 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:35 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:35.205+0000 7f2fbf0dc000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:40:35.292 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:35 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:35.293+0000 7f2fbf0dc000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:40:35.690 INFO:teuthology.orchestra.run.smithi092.stderr:mgr not available, waiting (2/15)... 2024-06-21T01:40:36.027 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:35 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:35.697+0000 7f2fbf0dc000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:40:36.028 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:35 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:35.827+0000 7f2fbf0dc000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:40:36.028 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:35 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:35.917+0000 7f2fbf0dc000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:40:36.028 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:36 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:36.029+0000 7f2fbf0dc000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:40:36.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:36 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:36.116+0000 7f2fbf0dc000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:40:37.011 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:36 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:36.655+0000 7f2fbf0dc000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:40:37.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:37.012+0000 7f2fbf0dc000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:40:37.820 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: Activating manager daemon y 2024-06-21T01:40:37.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: mgrmap e2: y(active, starting, since 0.00267481s) 2024-06-21T01:40:37.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: from='mgr.14100 172.21.15.92:0/2389017503' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:40:37.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: from='mgr.14100 172.21.15.92:0/2389017503' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:40:37.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: from='mgr.14100 172.21.15.92:0/2389017503' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:40:37.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: from='mgr.14100 172.21.15.92:0/2389017503' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:40:37.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: from='mgr.14100 172.21.15.92:0/2389017503' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:40:37.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: Manager daemon y is now available 2024-06-21T01:40:37.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: from='mgr.14100 172.21.15.92:0/2389017503' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:40:37.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: from='mgr.14100 172.21.15.92:0/2389017503' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:40:37.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: from='mgr.14100 172.21.15.92:0/2389017503' entity='mgr.y' 2024-06-21T01:40:37.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: from='mgr.14100 172.21.15.92:0/2389017503' entity='mgr.y' 2024-06-21T01:40:37.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:37 smithi092 ceph-mon[25107]: from='mgr.14100 172.21.15.92:0/2389017503' entity='mgr.y' 2024-06-21T01:40:38.778 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:38.778 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-06-21T01:40:38.778 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsid": "22eef780-2f6f-11ef-bc9d-c7b262605968", 2024-06-21T01:40:38.778 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "health": { 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "checks": {}, 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mutes": [] 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum": [ 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 0 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_names": [ 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "a" 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_age": 17, 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "monmap": { 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:38.779 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_mons": 1 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osdmap": { 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_osds": 0, 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgmap": { 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-06-21T01:40:38.780 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pools": 0, 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_objects": 0, 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_total": 0 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsmap": { 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "by_rank": [], 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "up:standby": 0 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mgrmap": { 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "available": true, 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-06-21T01:40:38.781 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modules": [ 2024-06-21T01:40:38.782 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "iostat", 2024-06-21T01:40:38.782 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "nfs", 2024-06-21T01:40:38.782 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "restful" 2024-06-21T01:40:38.782 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-06-21T01:40:38.782 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-06-21T01:40:38.782 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:38.782 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "servicemap": { 2024-06-21T01:40:38.782 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-21T01:40:38.782 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modified": "2024-06-21T01:40:16.829925+0000", 2024-06-21T01:40:38.782 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-06-21T01:40:38.783 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-06-21T01:40:38.783 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "progress_events": {} 2024-06-21T01:40:38.783 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-06-21T01:40:39.089 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:39 smithi092 ceph-mon[25107]: mgrmap e3: y(active, since 1.00695s) 2024-06-21T01:40:39.090 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:39 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1206520785' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-21T01:40:39.392 INFO:teuthology.orchestra.run.smithi092.stderr:mgr is available 2024-06-21T01:40:39.952 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:39.952 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [global] 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: fsid = 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [mgr] 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [osd] 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-06-21T01:40:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-06-21T01:40:40.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:40 smithi092 ceph-mon[25107]: mgrmap e4: y(active, since 2s) 2024-06-21T01:40:40.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:40 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/4159222089' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-21T01:40:40.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:40 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/4159222089' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-21T01:40:40.514 INFO:teuthology.orchestra.run.smithi092.stderr:Enabling cephadm module... 2024-06-21T01:40:41.476 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:41 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1528236550' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-21T01:40:41.477 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:41 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ignoring --setuser ceph since I am not root 2024-06-21T01:40:41.477 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:41 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ignoring --setgroup ceph since I am not root 2024-06-21T01:40:41.477 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:41 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:41.348+0000 7f353e445000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:40:41.477 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:41 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:41.479+0000 7f353e445000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:40:41.764 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:41 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:41.578+0000 7f353e445000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:40:42.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:41 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:41.892+0000 7f353e445000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:40:42.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:41 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:41.976+0000 7f353e445000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:40:42.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:42 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1528236550' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-21T01:40:42.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:42 smithi092 ceph-mon[25107]: mgrmap e5: y(active, since 4s) 2024-06-21T01:40:42.483 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-06-21T01:40:42.483 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 5, 2024-06-21T01:40:42.483 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "available": true, 2024-06-21T01:40:42.483 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "active_name": "y", 2024-06-21T01:40:42.483 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_standby": 0 2024-06-21T01:40:42.484 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-06-21T01:40:43.032 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for the mgr to restart... 2024-06-21T01:40:43.032 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mgr epoch 5... 2024-06-21T01:40:43.283 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:43 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/325483209' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-21T01:40:43.283 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:43.118+0000 7f353e445000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:40:43.283 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:43.285+0000 7f353e445000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:40:43.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:43.369+0000 7f353e445000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:40:44.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:43.685+0000 7f353e445000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:40:44.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:43.778+0000 7f353e445000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:40:44.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:43.871+0000 7f353e445000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:40:44.543 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:44 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:44.109+0000 7f353e445000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:40:44.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:44 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:44.546+0000 7f353e445000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:40:45.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:45 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:45.024+0000 7f353e445000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:40:45.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:45 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:45.116+0000 7f353e445000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:40:45.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:45 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:45.404+0000 7f353e445000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:40:45.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:45 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:45.492+0000 7f353e445000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:40:46.220 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:45 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:45.887+0000 7f353e445000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:40:46.220 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:46.017+0000 7f353e445000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:40:46.220 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:46.109+0000 7f353e445000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:40:46.220 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:46.222+0000 7f353e445000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:40:46.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:46.310+0000 7f353e445000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:40:47.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:46.847+0000 7f353e445000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:40:47.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:47 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:40:47.201+0000 7f353e445000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:40:47.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:47 smithi092 ceph-mon[25107]: Active manager daemon y restarted 2024-06-21T01:40:47.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:47 smithi092 ceph-mon[25107]: Activating manager daemon y 2024-06-21T01:40:47.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:47 smithi092 ceph-mon[25107]: osdmap e2: 0 total, 0 up, 0 in 2024-06-21T01:40:48.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:47 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:40:47] ENGINE Bus STARTING 2024-06-21T01:40:48.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:47 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:40:47] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:40:48.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:47 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:40:47] ENGINE Bus STARTED 2024-06-21T01:40:48.370 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-06-21T01:40:48.370 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-06-21T01:40:48.371 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "initialized": true 2024-06-21T01:40:48.371 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-06-21T01:40:48.651 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: mgrmap e6: y(active, starting, since 0.157896s) 2024-06-21T01:40:48.651 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: Manager daemon y is now available 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: [21/Jun/2024:01:40:47] ENGINE Bus STARTING 2024-06-21T01:40:48.652 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: [21/Jun/2024:01:40:47] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:40:48.653 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: [21/Jun/2024:01:40:47] ENGINE Bus STARTED 2024-06-21T01:40:48.653 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:40:48.653 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:48 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:40:48.947 INFO:teuthology.orchestra.run.smithi092.stderr:mgr epoch 5 is available 2024-06-21T01:40:48.948 INFO:teuthology.orchestra.run.smithi092.stderr:Setting orchestrator backend to cephadm... 2024-06-21T01:40:49.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:49 smithi092 ceph-mon[25107]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-21T01:40:49.533 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:49 smithi092 ceph-mon[25107]: mgrmap e7: y(active, since 1.16355s) 2024-06-21T01:40:49.533 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:49 smithi092 ceph-mon[25107]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-21T01:40:50.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:50 smithi092 ceph-mon[25107]: mgrmap e8: y(active, since 2s) 2024-06-21T01:40:50.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:50 smithi092 ceph-mon[25107]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:40:50.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:50 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:40:50.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:50 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:40:50.606 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: value unchanged 2024-06-21T01:40:51.151 INFO:teuthology.orchestra.run.smithi092.stderr:Generating ssh key... 2024-06-21T01:40:51.407 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-mon[25107]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:40:52.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: Generating public/private rsa key pair. 2024-06-21T01:40:52.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: Your identification has been saved in /tmp/tmpzak15qig/key. 2024-06-21T01:40:52.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: Your public key has been saved in /tmp/tmpzak15qig/key.pub. 2024-06-21T01:40:52.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: The key fingerprint is: 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: SHA256:22hvLoZJQVdkj4hj8JhAR/BhaNthVdhYnFY1+EmN+bs ceph-22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: The key's randomart image is: 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: +---[RSA 3072]----+ 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: | .++*..Bo==oo+ | 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: | o=o*+.*o.o+.. | 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: | . ++o=o. .o.o | 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: | . .... o . | 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: | .S . | 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: | . + . | 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: | . o+ . . | 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: | o.o.. E | 2024-06-21T01:40:52.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: | . +o | 2024-06-21T01:40:52.103 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:40:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: +----[SHA256]-----+ 2024-06-21T01:40:52.973 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHv54BaZZzEpaB14wiSvZvMF1IKil4VexpOWRD+wMG498DU1EaT9kfZqmEmMe9dZ6uCnYa2c0HZO0n1LFCr4UNmgTZH7Vh+UYQZmMh8jSGQAvSEZ8aP2vnp4XXsMXm+y86kUyK7XyQJ7HBJsjLVhCYM1BVbY4RwhsWnlSWoBOOlPaSIqKnpnmoqziPfSxFwJrljj1QpeDzv3IjXY6PsC1boj/fmX9x1Ga+nczO7r5Quwkk0rurBx3uufjpWCWAm6zTh9L5Yx0i8jS9PeJM3MXi9uBy7/WeaS5yhfH9HrO1VojzkptmhSyyJUNTG5Nh7pYEVohvdi2U/RjrDVEkBa6pJJkHs27p35eCRkDw/nBZmF2DZAWQhjC1+uUaKc2h9g99Iv7QylcX5OT5iGjMQkyb1wAsrcHMeEfcY0NWFe2OUoPETVfWyy+th5OwxgxlYIV7hxfFYcT3SGaTvQanQcLSrC7U5IKe4IPNyBLvMPB6UjDCK5lujWlNMujaqo9Out8= ceph-22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:40:53.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:52 smithi092 ceph-mon[25107]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:40:53.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:52 smithi092 ceph-mon[25107]: Generating ssh key... 2024-06-21T01:40:53.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:52 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:40:53.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:52 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:40:53.517 INFO:teuthology.orchestra.run.smithi092.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-21T01:40:53.518 INFO:teuthology.orchestra.run.smithi092.stderr:Adding key to root@localhost authorized_keys... 2024-06-21T01:40:53.518 INFO:teuthology.orchestra.run.smithi092.stderr:Adding host smithi092... 2024-06-21T01:40:54.081 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:53 smithi092 ceph-mon[25107]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:40:54.934 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: Added host 'smithi092' with addr '172.21.15.92' 2024-06-21T01:40:54.978 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:54 smithi092 ceph-mon[25107]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi092", "addr": "172.21.15.92", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:40:55.479 INFO:teuthology.orchestra.run.smithi092.stderr:Deploying unmanaged mon service... 2024-06-21T01:40:56.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:55 smithi092 ceph-mon[25107]: Deploying cephadm binary to smithi092 2024-06-21T01:40:56.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:55 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:40:56.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:55 smithi092 ceph-mon[25107]: Added host smithi092 2024-06-21T01:40:56.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:55 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:40:56.110 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: Scheduled mon update... 2024-06-21T01:40:56.811 INFO:teuthology.orchestra.run.smithi092.stderr:Deploying unmanaged mgr service... 2024-06-21T01:40:57.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:57 smithi092 ceph-mon[25107]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:40:57.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:57 smithi092 ceph-mon[25107]: Saving service mon spec with placement count:5 2024-06-21T01:40:57.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:57 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:40:57.522 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-06-21T01:40:58.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:58 smithi092 ceph-mon[25107]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:40:58.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:58 smithi092 ceph-mon[25107]: Saving service mgr spec with placement count:2 2024-06-21T01:40:58.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:40:58 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:41:00.221 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:00 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:41:00.221 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:00 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:41:00.221 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:00 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1978542541' entity='client.admin' 2024-06-21T01:41:01.912 INFO:teuthology.orchestra.run.smithi092.stderr:Enabling the dashboard module... 2024-06-21T01:41:02.135 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:01 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/4124454461' entity='client.admin' 2024-06-21T01:41:02.135 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:01 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:41:03.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:03 smithi092 ceph-mon[25107]: from='mgr.14118 172.21.15.92:0/3367942726' entity='mgr.y' 2024-06-21T01:41:03.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:03 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1907341399' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-21T01:41:03.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ignoring --setuser ceph since I am not root 2024-06-21T01:41:03.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ignoring --setgroup ceph since I am not root 2024-06-21T01:41:03.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:03.515+0000 7f5dee8a5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:41:04.049 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:03.648+0000 7f5dee8a5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:41:04.049 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:03.747+0000 7f5dee8a5000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:41:04.049 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:04 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:04.052+0000 7f5dee8a5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:41:04.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:04 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:04.135+0000 7f5dee8a5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:41:04.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:04 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1907341399' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-21T01:41:04.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:04 smithi092 ceph-mon[25107]: mgrmap e9: y(active, since 16s) 2024-06-21T01:41:04.857 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-06-21T01:41:04.857 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 9, 2024-06-21T01:41:04.857 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "available": true, 2024-06-21T01:41:04.857 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "active_name": "y", 2024-06-21T01:41:04.857 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_standby": 0 2024-06-21T01:41:04.857 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-06-21T01:41:05.431 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for the mgr to restart... 2024-06-21T01:41:05.432 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mgr epoch 9... 2024-06-21T01:41:05.540 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:05 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1311723143' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-21T01:41:05.540 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:05 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:05.290+0000 7f5dee8a5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:41:05.541 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:05 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:05.459+0000 7f5dee8a5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:41:05.541 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:05 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:05.543+0000 7f5dee8a5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:41:06.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:05 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:05.867+0000 7f5dee8a5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:41:06.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:05 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:05.959+0000 7f5dee8a5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:41:06.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:06 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:06.050+0000 7f5dee8a5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:41:06.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:06 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:06.289+0000 7f5dee8a5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:41:07.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:06 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:06.723+0000 7f5dee8a5000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:41:07.584 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:07 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:07.206+0000 7f5dee8a5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:41:07.584 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:07 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:07.298+0000 7f5dee8a5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:41:07.584 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:07 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:07.587+0000 7f5dee8a5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:41:07.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:07 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:07.675+0000 7f5dee8a5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:41:08.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:08 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:08.073+0000 7f5dee8a5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:41:08.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:08 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:08.204+0000 7f5dee8a5000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:41:08.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:08 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:08.295+0000 7f5dee8a5000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:41:08.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:08 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:08.408+0000 7f5dee8a5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:41:08.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:08 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:08.497+0000 7f5dee8a5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:41:09.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:09.036+0000 7f5dee8a5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:41:09.759 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:41:09.391+0000 7f5dee8a5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:41:09.759 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:41:09] ENGINE Bus STARTING 2024-06-21T01:41:10.077 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:41:09] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:41:10.077 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:41:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:41:09] ENGINE Bus STARTED 2024-06-21T01:41:10.457 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-06-21T01:41:10.457 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-06-21T01:41:10.457 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "initialized": true 2024-06-21T01:41:10.457 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-06-21T01:41:10.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: Active manager daemon y restarted 2024-06-21T01:41:10.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: Activating manager daemon y 2024-06-21T01:41:10.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: osdmap e3: 0 total, 0 up, 0 in 2024-06-21T01:41:10.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: mgrmap e10: y(active, starting, since 0.0552853s) 2024-06-21T01:41:10.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:41:10.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:41:10.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:41:10.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:41:10.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:41:10.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: Manager daemon y is now available 2024-06-21T01:41:10.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:41:10.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:41:10.823 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:41:10.823 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:41:10.823 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:11.124 INFO:teuthology.orchestra.run.smithi092.stderr:mgr epoch 9 is available 2024-06-21T01:41:11.124 INFO:teuthology.orchestra.run.smithi092.stderr:Generating a dashboard self-signed certificate... 2024-06-21T01:41:11.707 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:11 smithi092 ceph-mon[25107]: [21/Jun/2024:01:41:09] ENGINE Bus STARTING 2024-06-21T01:41:11.707 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:11 smithi092 ceph-mon[25107]: [21/Jun/2024:01:41:09] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:41:11.707 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:11 smithi092 ceph-mon[25107]: [21/Jun/2024:01:41:09] ENGINE Bus STARTED 2024-06-21T01:41:11.707 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:11 smithi092 ceph-mon[25107]: mgrmap e11: y(active, since 1.05966s) 2024-06-21T01:41:11.798 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: Self-signed certificate created 2024-06-21T01:41:12.414 INFO:teuthology.orchestra.run.smithi092.stderr:Creating initial admin user... 2024-06-21T01:41:12.644 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:12 smithi092 ceph-mon[25107]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-21T01:41:12.644 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:12 smithi092 ceph-mon[25107]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-21T01:41:12.644 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:12 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:12.644 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:12 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:12.644 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:12 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:12.644 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:12 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:12.644 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:12 smithi092 ceph-mon[25107]: mgrmap e12: y(active, since 2s) 2024-06-21T01:41:13.294 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$eIUuIjog0qex8prU2G2biu8BxUEY2b/QFUoVTBsiXVgKMjdL2lyqS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1718934073, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-21T01:41:13.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:13 smithi092 ceph-mon[25107]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:41:13.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:13 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:14.075 INFO:teuthology.orchestra.run.smithi092.stderr:Fetching dashboard port number... 2024-06-21T01:41:14.677 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:14 smithi092 ceph-mon[25107]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:41:14.710 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 8443 2024-06-21T01:41:15.665 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:15 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2057516372' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-21T01:41:15.678 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: disabled 2024-06-21T01:41:15.686 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-06-21T01:41:15.687 INFO:teuthology.orchestra.run.smithi092.stderr:firewalld.service is not enabled 2024-06-21T01:41:15.688 INFO:teuthology.orchestra.run.smithi092.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-21T01:41:15.690 INFO:teuthology.orchestra.run.smithi092.stderr:Ceph Dashboard is now available at: 2024-06-21T01:41:15.690 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-06-21T01:41:15.690 INFO:teuthology.orchestra.run.smithi092.stderr: URL: https://smithi092.front.sepia.ceph.com:8443/ 2024-06-21T01:41:15.690 INFO:teuthology.orchestra.run.smithi092.stderr: User: admin 2024-06-21T01:41:15.690 INFO:teuthology.orchestra.run.smithi092.stderr: Password: d4jh8n0716 2024-06-21T01:41:15.691 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-06-21T01:41:15.691 INFO:teuthology.orchestra.run.smithi092.stderr:Enabling autotune for osd_memory_target 2024-06-21T01:41:16.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:16 smithi092 ceph-mon[25107]: mgrmap e13: y(active, since 6s) 2024-06-21T01:41:17.671 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-06-21T01:41:18.247 INFO:teuthology.orchestra.run.smithi092.stderr:You can access the Ceph CLI with: 2024-06-21T01:41:18.248 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-06-21T01:41:18.248 INFO:teuthology.orchestra.run.smithi092.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-21T01:41:18.248 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-06-21T01:41:18.248 INFO:teuthology.orchestra.run.smithi092.stderr:Please consider enabling telemetry to help improve Ceph: 2024-06-21T01:41:18.249 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-06-21T01:41:18.249 INFO:teuthology.orchestra.run.smithi092.stderr: ceph telemetry on 2024-06-21T01:41:18.249 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-06-21T01:41:18.249 INFO:teuthology.orchestra.run.smithi092.stderr:For more information see: 2024-06-21T01:41:18.249 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-06-21T01:41:18.249 INFO:teuthology.orchestra.run.smithi092.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-06-21T01:41:18.250 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-06-21T01:41:18.250 INFO:teuthology.orchestra.run.smithi092.stderr:Bootstrap complete. 2024-06-21T01:41:18.283 INFO:tasks.cephadm:Fetching config... 2024-06-21T01:41:18.284 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-06-21T01:41:18.284 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-21T01:41:18.300 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-21T01:41:18.300 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-06-21T01:41:18.300 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-21T01:41:18.356 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-21T01:41:18.357 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-06-21T01:41:18.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/keyring of=/dev/stdout 2024-06-21T01:41:18.421 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-21T01:41:18.421 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-06-21T01:41:18.421 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-21T01:41:18.477 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-21T01:41:18.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHv54BaZZzEpaB14wiSvZvMF1IKil4VexpOWRD+wMG498DU1EaT9kfZqmEmMe9dZ6uCnYa2c0HZO0n1LFCr4UNmgTZH7Vh+UYQZmMh8jSGQAvSEZ8aP2vnp4XXsMXm+y86kUyK7XyQJ7HBJsjLVhCYM1BVbY4RwhsWnlSWoBOOlPaSIqKnpnmoqziPfSxFwJrljj1QpeDzv3IjXY6PsC1boj/fmX9x1Ga+nczO7r5Quwkk0rurBx3uufjpWCWAm6zTh9L5Yx0i8jS9PeJM3MXi9uBy7/WeaS5yhfH9HrO1VojzkptmhSyyJUNTG5Nh7pYEVohvdi2U/RjrDVEkBa6pJJkHs27p35eCRkDw/nBZmF2DZAWQhjC1+uUaKc2h9g99Iv7QylcX5OT5iGjMQkyb1wAsrcHMeEfcY0NWFe2OUoPETVfWyy+th5OwxgxlYIV7hxfFYcT3SGaTvQanQcLSrC7U5IKe4IPNyBLvMPB6UjDCK5lujWlNMujaqo9Out8= ceph-22eef780-2f6f-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-21T01:41:18.553 INFO:teuthology.orchestra.run.smithi092.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHv54BaZZzEpaB14wiSvZvMF1IKil4VexpOWRD+wMG498DU1EaT9kfZqmEmMe9dZ6uCnYa2c0HZO0n1LFCr4UNmgTZH7Vh+UYQZmMh8jSGQAvSEZ8aP2vnp4XXsMXm+y86kUyK7XyQJ7HBJsjLVhCYM1BVbY4RwhsWnlSWoBOOlPaSIqKnpnmoqziPfSxFwJrljj1QpeDzv3IjXY6PsC1boj/fmX9x1Ga+nczO7r5Quwkk0rurBx3uufjpWCWAm6zTh9L5Yx0i8jS9PeJM3MXi9uBy7/WeaS5yhfH9HrO1VojzkptmhSyyJUNTG5Nh7pYEVohvdi2U/RjrDVEkBa6pJJkHs27p35eCRkDw/nBZmF2DZAWQhjC1+uUaKc2h9g99Iv7QylcX5OT5iGjMQkyb1wAsrcHMeEfcY0NWFe2OUoPETVfWyy+th5OwxgxlYIV7hxfFYcT3SGaTvQanQcLSrC7U5IKe4IPNyBLvMPB6UjDCK5lujWlNMujaqo9Out8= ceph-22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:41:18.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHv54BaZZzEpaB14wiSvZvMF1IKil4VexpOWRD+wMG498DU1EaT9kfZqmEmMe9dZ6uCnYa2c0HZO0n1LFCr4UNmgTZH7Vh+UYQZmMh8jSGQAvSEZ8aP2vnp4XXsMXm+y86kUyK7XyQJ7HBJsjLVhCYM1BVbY4RwhsWnlSWoBOOlPaSIqKnpnmoqziPfSxFwJrljj1QpeDzv3IjXY6PsC1boj/fmX9x1Ga+nczO7r5Quwkk0rurBx3uufjpWCWAm6zTh9L5Yx0i8jS9PeJM3MXi9uBy7/WeaS5yhfH9HrO1VojzkptmhSyyJUNTG5Nh7pYEVohvdi2U/RjrDVEkBa6pJJkHs27p35eCRkDw/nBZmF2DZAWQhjC1+uUaKc2h9g99Iv7QylcX5OT5iGjMQkyb1wAsrcHMeEfcY0NWFe2OUoPETVfWyy+th5OwxgxlYIV7hxfFYcT3SGaTvQanQcLSrC7U5IKe4IPNyBLvMPB6UjDCK5lujWlNMujaqo9Out8= ceph-22eef780-2f6f-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-21T01:41:18.609 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHv54BaZZzEpaB14wiSvZvMF1IKil4VexpOWRD+wMG498DU1EaT9kfZqmEmMe9dZ6uCnYa2c0HZO0n1LFCr4UNmgTZH7Vh+UYQZmMh8jSGQAvSEZ8aP2vnp4XXsMXm+y86kUyK7XyQJ7HBJsjLVhCYM1BVbY4RwhsWnlSWoBOOlPaSIqKnpnmoqziPfSxFwJrljj1QpeDzv3IjXY6PsC1boj/fmX9x1Ga+nczO7r5Quwkk0rurBx3uufjpWCWAm6zTh9L5Yx0i8jS9PeJM3MXi9uBy7/WeaS5yhfH9HrO1VojzkptmhSyyJUNTG5Nh7pYEVohvdi2U/RjrDVEkBa6pJJkHs27p35eCRkDw/nBZmF2DZAWQhjC1+uUaKc2h9g99Iv7QylcX5OT5iGjMQkyb1wAsrcHMeEfcY0NWFe2OUoPETVfWyy+th5OwxgxlYIV7hxfFYcT3SGaTvQanQcLSrC7U5IKe4IPNyBLvMPB6UjDCK5lujWlNMujaqo9Out8= ceph-22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:41:18.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-21T01:41:18.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:18 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1885716335' entity='client.admin' 2024-06-21T01:41:22.131 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-21T01:41:22.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-21T01:41:22.678 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:22 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1318838602' entity='client.admin' 2024-06-21T01:41:23.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:23 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:23.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:23 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:41:23.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:23 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:23.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:23 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:41:23.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:23 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:24.998 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-06-21T01:41:24.999 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-21T01:41:24.999 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-06-21T01:41:25.019 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-21T01:41:25.019 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:41:25.078 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-06-21T01:41:25.079 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch host add smithi173 2024-06-21T01:41:25.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:25 smithi092 ceph-mon[25107]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:41:25.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:25 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:25.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:25 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:41:25.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:25 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:25.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:25 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:41:25.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:25 smithi092 ceph-mon[25107]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:41:25.599 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:25 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:25.599 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:25 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:26.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:26 smithi092 ceph-mon[25107]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:41:27.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:27 smithi092 ceph-mon[25107]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:41:27.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:27 smithi092 ceph-mon[25107]: Deploying cephadm binary to smithi173 2024-06-21T01:41:27.874 INFO:teuthology.orchestra.run.smithi092.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-06-21T01:41:28.444 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch host ls --format=json 2024-06-21T01:41:28.991 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:28 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:28.991 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:28 smithi092 ceph-mon[25107]: Added host smithi173 2024-06-21T01:41:28.991 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:28 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:41:28.991 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:28 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:28.991 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:28 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:41:28.991 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:28 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:28.992 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:28 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:30.267 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:41:30.267 INFO:teuthology.orchestra.run.smithi092.stdout:[{"addr": "172.21.15.92", "hostname": "smithi092", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-06-21T01:41:30.549 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:30 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:30.549 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:30 smithi092 ceph-mon[25107]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:30.549 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:30 smithi092 ceph-mon[25107]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-21T01:41:30.789 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-21T01:41:30.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd crush tunables default 2024-06-21T01:41:33.372 INFO:teuthology.orchestra.run.smithi092.stderr:adjusted tunables profile to default 2024-06-21T01:41:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:33 smithi092 ceph-mon[25107]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:33 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:33 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:33 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:41:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:33 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:33 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1935020875' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-21T01:41:34.009 INFO:tasks.cephadm:Adding mon.a on smithi092 2024-06-21T01:41:34.009 INFO:tasks.cephadm:Adding mon.c on smithi092 2024-06-21T01:41:34.009 INFO:tasks.cephadm:Adding mon.b on smithi173 2024-06-21T01:41:34.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch apply mon '3;smithi092:172.21.15.92=a;smithi092:[v2:172.21.15.92:3301,v1:172.21.15.92:6790]=c;smithi173:172.21.15.173=b' 2024-06-21T01:41:34.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:34 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1935020875' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-21T01:41:34.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:34 smithi092 ceph-mon[25107]: osdmap e4: 0 total, 0 up, 0 in 2024-06-21T01:41:35.706 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-06-21T01:41:35.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:35 smithi092 ceph-mon[25107]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:36.384 DEBUG:teuthology.orchestra.run.smithi092:mon.c> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.c.service 2024-06-21T01:41:36.387 DEBUG:teuthology.orchestra.run.smithi173:mon.b> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.b.service 2024-06-21T01:41:36.389 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-21T01:41:36.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-21T01:41:37.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:37.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi092:172.21.15.92=a;smithi092:[v2:172.21.15.92:3301,v1:172.21.15.92:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: Saving service mon spec with placement smithi092:172.21.15.92=a;smithi092:[v2:172.21.15.92:3301,v1:172.21.15.92:6790]=c;smithi173:172.21.15.173=b;count:3 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:37.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:36 smithi092 ceph-mon[25107]: Deploying daemon mon.c on smithi092 2024-06-21T01:41:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-21T01:41:38.170 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"22eef780-2f6f-11ef-bc9d-c7b262605968","modified":"2024-06-21T01:40:13.962891Z","created":"2024-06-21T01:40:13.962891Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3300","nonce":0},{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-21T01:41:38.171 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-21T01:41:38.958 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:38 smithi092 ceph-mon[25107]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:38.958 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:38 smithi092 ceph-mon[25107]: from='client.? 172.21.15.173:0/2663086004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-21T01:41:39.765 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-21T01:41:39.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-21T01:41:44.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:41:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: mon.a calling monitor election 2024-06-21T01:41:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:41:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:41:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:41:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: mon.c calling monitor election 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: monmap e2: 2 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0]} 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: fsmap 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: osdmap e4: 0 total, 0 up, 0 in 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: mgrmap e13: y(active, since 34s) 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: overall HEALTH_OK 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:41:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:44.697 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-21T01:41:44.698 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"22eef780-2f6f-11ef-bc9d-c7b262605968","modified":"2024-06-21T01:41:39.139748Z","created":"2024-06-21T01:40:13.962891Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3300","nonce":0},{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3301","nonce":0},{"type":"v1","addr":"172.21.15.92:6790","nonce":0}]},"addr":"172.21.15.92:6790/0","public_addr":"172.21.15.92:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-21T01:41:44.700 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-06-21T01:41:45.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:45 smithi092 ceph-mon[25107]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:45.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:45 smithi092 ceph-mon[25107]: Deploying daemon mon.b on smithi173 2024-06-21T01:41:45.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:45 smithi092 ceph-mon[25107]: from='client.? 172.21.15.173:0/2022187334' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-21T01:41:45.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:45 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:41:45.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:45 smithi092 ceph-mon[29810]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:45.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:45 smithi092 ceph-mon[29810]: Deploying daemon mon.b on smithi173 2024-06-21T01:41:45.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:45 smithi092 ceph-mon[29810]: from='client.? 172.21.15.173:0/2022187334' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-21T01:41:45.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:45 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:41:46.867 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-21T01:41:46.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-21T01:41:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:47 smithi092 ceph-mon[25107]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:47.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:47 smithi092 ceph-mon[29810]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-21T01:41:49.023 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"22eef780-2f6f-11ef-bc9d-c7b262605968","modified":"2024-06-21T01:41:39.139748Z","created":"2024-06-21T01:40:13.962891Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3300","nonce":0},{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3301","nonce":0},{"type":"v1","addr":"172.21.15.92:6790","nonce":0}]},"addr":"172.21.15.92:6790/0","public_addr":"172.21.15.92:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-21T01:41:49.025 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-06-21T01:41:50.814 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-21T01:41:50.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-21T01:41:54.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:41:54.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: mon.a calling monitor election 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: mon.c calling monitor election 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: monmap e3: 3 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0]} 2024-06-21T01:41:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: fsmap 2024-06-21T01:41:54.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: osdmap e4: 0 total, 0 up, 0 in 2024-06-21T01:41:54.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: mgrmap e13: y(active, since 44s) 2024-06-21T01:41:54.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-06-21T01:41:54.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-06-21T01:41:54.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-06-21T01:41:54.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: mon.b (rank 2) addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] is down (out of quorum) 2024-06-21T01:41:54.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:54.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:41:54.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:54.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:41:54.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: mon.a calling monitor election 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: mon.c calling monitor election 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: monmap e3: 3 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0]} 2024-06-21T01:41:54.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: fsmap 2024-06-21T01:41:54.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: osdmap e4: 0 total, 0 up, 0 in 2024-06-21T01:41:54.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: mgrmap e13: y(active, since 44s) 2024-06-21T01:41:54.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-06-21T01:41:54.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-06-21T01:41:54.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-06-21T01:41:54.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: mon.b (rank 2) addr [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] is down (out of quorum) 2024-06-21T01:41:54.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:54.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:41:54.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:54.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:41:55.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:55 smithi092 ceph-mon[29810]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:55.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:55 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:55.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:55 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:55.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:55 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:55.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:55 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:55.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:55 smithi092 ceph-mon[25107]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:55.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:55 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:55.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:55 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:55.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:55 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:55.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:55 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:55.798 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-21T01:41:55.799 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"22eef780-2f6f-11ef-bc9d-c7b262605968","modified":"2024-06-21T01:41:49.138463Z","created":"2024-06-21T01:40:13.962891Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3300","nonce":0},{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3301","nonce":0},{"type":"v1","addr":"172.21.15.92:6790","nonce":0}]},"addr":"172.21.15.92:6790/0","public_addr":"172.21.15.92:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-21T01:41:55.801 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2024-06-21T01:41:56.377 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-21T01:41:56.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph config generate-minimal-conf 2024-06-21T01:41:56.635 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: mon.b calling monitor election 2024-06-21T01:41:56.635 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: mon.b calling monitor election 2024-06-21T01:41:56.635 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: mon.c calling monitor election 2024-06-21T01:41:56.635 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: mon.a calling monitor election 2024-06-21T01:41:56.635 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-21T01:41:56.635 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: monmap e3: 3 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0]} 2024-06-21T01:41:56.635 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: fsmap 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: osdmap e4: 0 total, 0 up, 0 in 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: mgrmap e13: y(active, since 46s) 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: Cluster is now healthy 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[29810]: overall HEALTH_OK 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: mon.b calling monitor election 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: mon.b calling monitor election 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: mon.c calling monitor election 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: mon.a calling monitor election 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: monmap e3: 3 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0]} 2024-06-21T01:41:56.636 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: fsmap 2024-06-21T01:41:56.637 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: osdmap e4: 0 total, 0 up, 0 in 2024-06-21T01:41:56.637 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: mgrmap e13: y(active, since 46s) 2024-06-21T01:41:56.637 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-06-21T01:41:56.637 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: Cluster is now healthy 2024-06-21T01:41:56.637 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:56 smithi092 ceph-mon[25107]: overall HEALTH_OK 2024-06-21T01:41:57.724 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[25107]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:57.724 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:57.724 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:41:57.724 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:41:57.724 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:57.724 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:57.724 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[29810]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:57.724 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:57.724 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:41:57.725 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:41:57.725 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:57.725 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:57.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:41:57 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:41:58.539 INFO:teuthology.orchestra.run.smithi092.stdout:# minimal ceph.conf for 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:41:58.539 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2024-06-21T01:41:58.539 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:41:58.539 INFO:teuthology.orchestra.run.smithi092.stdout: mon_host = [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] 2024-06-21T01:41:58.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:58 smithi092 ceph-mon[25107]: Reconfiguring mon.c (monmap changed)... 2024-06-21T01:41:58.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:58 smithi092 ceph-mon[25107]: Reconfiguring daemon mon.c on smithi092 2024-06-21T01:41:58.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:58 smithi092 ceph-mon[29810]: Reconfiguring mon.c (monmap changed)... 2024-06-21T01:41:58.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:58 smithi092 ceph-mon[29810]: Reconfiguring daemon mon.c on smithi092 2024-06-21T01:41:58.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:41:58 smithi173 ceph-mon[25952]: Reconfiguring mon.c (monmap changed)... 2024-06-21T01:41:58.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:41:58 smithi173 ceph-mon[25952]: Reconfiguring daemon mon.c on smithi092 2024-06-21T01:41:59.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[25107]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:59.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3129448898' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:59.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:59.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:41:59.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:41:59.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:59.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[29810]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:59.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3129448898' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:59.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:59.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:41:59.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:41:59.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:41:59 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:59.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:41:59 smithi173 ceph-mon[25952]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:41:59.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:41:59 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3129448898' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:59.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:41:59 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:41:59.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:41:59 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:41:59.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:41:59 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:41:59.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:41:59 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:41:59.809 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-21T01:41:59.810 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-06-21T01:41:59.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.conf 2024-06-21T01:41:59.837 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-06-21T01:41:59.837 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:41:59.903 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-21T01:41:59.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-06-21T01:41:59.928 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-21T01:41:59.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:41:59.993 INFO:tasks.cephadm:Adding mgr.y on smithi092 2024-06-21T01:41:59.993 INFO:tasks.cephadm:Adding mgr.x on smithi173 2024-06-21T01:41:59.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch apply mgr '2;smithi092=y;smithi173=x' 2024-06-21T01:42:00.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:00 smithi173 ceph-mon[25952]: Reconfiguring mon.b (monmap changed)... 2024-06-21T01:42:00.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:00 smithi173 ceph-mon[25952]: Reconfiguring daemon mon.b on smithi173 2024-06-21T01:42:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:00 smithi092 ceph-mon[25107]: Reconfiguring mon.b (monmap changed)... 2024-06-21T01:42:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:00 smithi092 ceph-mon[25107]: Reconfiguring daemon mon.b on smithi173 2024-06-21T01:42:00.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:00 smithi092 ceph-mon[29810]: Reconfiguring mon.b (monmap changed)... 2024-06-21T01:42:00.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:00 smithi092 ceph-mon[29810]: Reconfiguring daemon mon.b on smithi173 2024-06-21T01:42:01.570 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:01 smithi173 ceph-mon[25952]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:01.570 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:01 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:01.570 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:01 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:01.570 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:01 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:01.570 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:01 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:01.570 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:01 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:01.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[25107]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:01.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:01.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:01.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:01.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:01.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:01.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[29810]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:01.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:01.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:01.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:01.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:01.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:01 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:02.194 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2024-06-21T01:42:02.774 DEBUG:teuthology.orchestra.run.smithi173:mgr.x> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.x.service 2024-06-21T01:42:02.777 INFO:tasks.cephadm:Deploying OSDs... 2024-06-21T01:42:02.777 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-06-21T01:42:02.778 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2024-06-21T01:42:02.797 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-21T01:42:02.798 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2024-06-21T01:42:02.855 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-21T01:42:02.856 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-21T01:42:02.856 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-06-21T01:42:02.856 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-21T01:42:02.856 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-06-21T01:42:02.856 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-06-21 01:41:38.590168172 +0000 2024-06-21T01:42:02.856 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-06-21 01:38:58.747390875 +0000 2024-06-21T01:42:02.856 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-06-21 01:38:58.747390875 +0000 2024-06-21T01:42:02.856 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-06-21 01:38:58.747390875 +0000 2024-06-21T01:42:02.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-21T01:42:02.921 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-06-21T01:42:02.921 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-06-21T01:42:02.921 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000262274 s, 2.0 MB/s 2024-06-21T01:42:02.922 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-21T01:42:02.980 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2024-06-21T01:42:03.037 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-21T01:42:03.037 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-21T01:42:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 692 Links: 1 2024-06-21T01:42:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-21T01:42:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-06-21T01:42:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-06-21 01:41:38.591168144 +0000 2024-06-21T01:42:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-06-21 01:38:59.070381867 +0000 2024-06-21T01:42:03.038 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-06-21 01:38:59.070381867 +0000 2024-06-21T01:42:03.038 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-06-21 01:38:59.070381867 +0000 2024-06-21T01:42:03.038 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-21T01:42:03.102 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-06-21T01:42:03.102 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-06-21T01:42:03.102 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000204078 s, 2.5 MB/s 2024-06-21T01:42:03.103 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-21T01:42:03.161 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2024-06-21T01:42:03.218 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-21T01:42:03.218 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-21T01:42:03.218 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 712 Links: 1 2024-06-21T01:42:03.218 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-21T01:42:03.219 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-06-21T01:42:03.219 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-06-21 01:41:38.591168144 +0000 2024-06-21T01:42:03.219 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-06-21 01:38:59.418372162 +0000 2024-06-21T01:42:03.219 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-06-21 01:38:59.418372162 +0000 2024-06-21T01:42:03.219 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-06-21 01:38:59.418372162 +0000 2024-06-21T01:42:03.219 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-21T01:42:03.286 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-06-21T01:42:03.286 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-06-21T01:42:03.286 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000184423 s, 2.8 MB/s 2024-06-21T01:42:03.287 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-21T01:42:03.322 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:03.323 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi092=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:03.323 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: Saving service mgr spec with placement smithi092=y;smithi173=x;count:2 2024-06-21T01:42:03.323 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:03.323 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:03.323 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:03.323 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:03.323 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:03.323 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:42:03.323 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-21T01:42:03.324 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:42:03.324 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:03.324 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:03 smithi173 ceph-mon[25952]: Deploying daemon mgr.x on smithi173 2024-06-21T01:42:03.343 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2024-06-21T01:42:03.399 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-21T01:42:03.399 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-21T01:42:03.399 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 729 Links: 1 2024-06-21T01:42:03.399 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-21T01:42:03.399 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-06-21T01:42:03.399 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-06-21 01:41:38.591168144 +0000 2024-06-21T01:42:03.399 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-06-21 01:38:59.736363293 +0000 2024-06-21T01:42:03.400 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-06-21 01:38:59.736363293 +0000 2024-06-21T01:42:03.400 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-06-21 01:38:59.736363293 +0000 2024-06-21T01:42:03.400 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-21T01:42:03.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:03.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi092=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:03.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: Saving service mgr spec with placement smithi092=y;smithi173=x;count:2 2024-06-21T01:42:03.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:03.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:03.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:03.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:03.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:03.462 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:42:03.462 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-21T01:42:03.462 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:42:03.462 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:03.462 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[25107]: Deploying daemon mgr.x on smithi173 2024-06-21T01:42:03.462 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:03.462 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi092=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:03.462 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: Saving service mgr spec with placement smithi092=y;smithi173=x;count:2 2024-06-21T01:42:03.462 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:03.462 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:03.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:03.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:03.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:03.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:42:03.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-21T01:42:03.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:42:03.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:03.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:03 smithi092 ceph-mon[29810]: Deploying daemon mgr.x on smithi173 2024-06-21T01:42:03.463 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-06-21T01:42:03.464 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-06-21T01:42:03.464 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000188799 s, 2.7 MB/s 2024-06-21T01:42:03.464 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-21T01:42:03.521 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-21T01:42:03.521 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-06-21T01:42:03.537 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-21T01:42:03.538 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-06-21T01:42:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-21T01:42:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-21T01:42:03.593 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-06-21T01:42:03.593 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-21T01:42:03.594 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-06-21T01:42:03.594 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-06-21 01:41:47.705403901 +0000 2024-06-21T01:42:03.594 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-06-21 01:38:58.264333724 +0000 2024-06-21T01:42:03.594 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-06-21 01:38:58.264333724 +0000 2024-06-21T01:42:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-06-21 01:38:58.264333724 +0000 2024-06-21T01:42:03.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-21T01:42:03.658 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-06-21T01:42:03.658 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-06-21T01:42:03.658 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000207773 s, 2.5 MB/s 2024-06-21T01:42:03.659 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-21T01:42:03.716 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-06-21T01:42:03.773 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-21T01:42:03.773 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-21T01:42:03.773 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-06-21T01:42:03.773 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-21T01:42:03.773 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-06-21T01:42:03.773 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-06-21 01:41:47.705403901 +0000 2024-06-21T01:42:03.773 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-06-21 01:38:58.615326256 +0000 2024-06-21T01:42:03.774 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-06-21 01:38:58.615326256 +0000 2024-06-21T01:42:03.774 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-06-21 01:38:58.615326256 +0000 2024-06-21T01:42:03.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-21T01:42:03.837 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-06-21T01:42:03.838 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-06-21T01:42:03.838 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000200866 s, 2.5 MB/s 2024-06-21T01:42:03.839 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-21T01:42:03.895 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-06-21T01:42:03.951 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-21T01:42:03.952 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-21T01:42:03.952 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-06-21T01:42:03.952 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-21T01:42:03.952 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-06-21T01:42:03.952 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-06-21 01:41:47.706403880 +0000 2024-06-21T01:42:03.953 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-06-21 01:38:58.953319064 +0000 2024-06-21T01:42:03.953 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-06-21 01:38:58.953319064 +0000 2024-06-21T01:42:03.953 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-06-21 01:38:58.953319064 +0000 2024-06-21T01:42:03.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-21T01:42:04.151 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-06-21T01:42:04.151 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-06-21T01:42:04.151 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000225212 s, 2.3 MB/s 2024-06-21T01:42:04.152 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-21T01:42:04.169 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-06-21T01:42:04.226 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-21T01:42:04.226 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-21T01:42:04.227 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-06-21T01:42:04.227 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-21T01:42:04.227 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-06-21T01:42:04.227 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-06-21 01:42:04.111058440 +0000 2024-06-21T01:42:04.227 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-06-21 01:38:59.298311724 +0000 2024-06-21T01:42:04.227 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-06-21 01:38:59.298311724 +0000 2024-06-21T01:42:04.227 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-06-21 01:38:59.298311724 +0000 2024-06-21T01:42:04.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-21T01:42:04.322 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-06-21T01:42:04.322 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-06-21T01:42:04.322 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000169489 s, 3.0 MB/s 2024-06-21T01:42:04.323 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-21T01:42:04.338 INFO:tasks.cephadm:Deploying osd.0 on smithi092 with /dev/vg_nvme/lv_4... 2024-06-21T01:42:04.339 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-21T01:42:04.564 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:04 smithi173 systemd[1]: Starting Ceph mgr.x for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:42:04.833 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:04 smithi173 podman[27114]: 2024-06-21 01:42:04.56705724 +0000 UTC m=+0.111360483 container create 8ba723b662bcd4df1a845f7beca94040761c279e4d998f79b74ea6c1307dab58 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, version=8, architecture=x86_64, distribution-scope=public, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-06-21T01:42:04.833 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:04 smithi173 podman[27114]: 2024-06-21 01:42:04.473587572 +0000 UTC m=+0.017890825 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-06-21T01:42:04.833 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:04 smithi173 podman[27114]: 2024-06-21 01:42:04.811743733 +0000 UTC m=+0.356046989 container init 8ba723b662bcd4df1a845f7beca94040761c279e4d998f79b74ea6c1307dab58 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-type=git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, name=centos-stream, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64) 2024-06-21T01:42:04.833 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:04 smithi173 podman[27114]: 2024-06-21 01:42:04.815066827 +0000 UTC m=+0.359370072 container start 8ba723b662bcd4df1a845f7beca94040761c279e4d998f79b74ea6c1307dab58 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, io.openshift.expose-services=, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, version=8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container) 2024-06-21T01:42:05.091 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:04 smithi173 ceph-mgr[27133]: -- 172.21.15.173:0/709384971 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560e9e01a340 con 0x560e9dff2800 2024-06-21T01:42:05.091 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:04 smithi173 bash[27114]: 8ba723b662bcd4df1a845f7beca94040761c279e4d998f79b74ea6c1307dab58 2024-06-21T01:42:05.091 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:04 smithi173 systemd[1]: Started Ceph mgr.x for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:42:05.091 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:04 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:04.959+0000 7f6f305de000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:42:05.091 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:05.094+0000 7f6f305de000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:42:05.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:05.211+0000 7f6f305de000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:42:05.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:05 smithi173 ceph-mon[25952]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:05.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:05.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:05.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:05.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:05.568 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:05 smithi092 ceph-mon[25107]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:05.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:05.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:05.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:05.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:05.569 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:05 smithi092 ceph-mon[29810]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:05.569 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:05.569 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:05.569 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:05.569 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:05.977 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:05.532+0000 7f6f305de000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:42:05.977 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:05.619+0000 7f6f305de000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:42:06.323 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-21T01:42:06.323 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-21T01:42:06.323 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-06-21T01:42:06.323 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-06-21T01:42:06.323 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119517 s, 877 MB/s 2024-06-21T01:42:06.323 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-06-21T01:42:07.002 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_4 2024-06-21T01:42:07.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[25107]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:07.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:07.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:07.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[25107]: Reconfiguring mgr.y (unknown last config time)... 2024-06-21T01:42:07.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:42:07.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:42:07.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:07.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[25107]: Reconfiguring daemon mgr.y on smithi092 2024-06-21T01:42:07.029 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[29810]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:07.029 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:07.029 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:07.029 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[29810]: Reconfiguring mgr.y (unknown last config time)... 2024-06-21T01:42:07.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:42:07.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:42:07.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:07.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:06 smithi092 ceph-mon[29810]: Reconfiguring daemon mgr.y on smithi092 2024-06-21T01:42:07.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:06.764+0000 7f6f305de000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:42:07.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:06.925+0000 7f6f305de000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:42:07.229 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:07.006+0000 7f6f305de000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:42:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:06 smithi173 ceph-mon[25952]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:06 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:06 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:06 smithi173 ceph-mon[25952]: Reconfiguring mgr.y (unknown last config time)... 2024-06-21T01:42:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:06 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:42:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:06 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:42:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:06 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:06 smithi173 ceph-mon[25952]: Reconfiguring daemon mgr.y on smithi092 2024-06-21T01:42:07.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:07.317+0000 7f6f305de000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:42:07.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:07.406+0000 7f6f305de000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:42:07.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:07.498+0000 7f6f305de000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:42:08.166 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:07.736+0000 7f6f305de000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:42:08.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:08.167+0000 7f6f305de000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:42:08.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:08.650+0000 7f6f305de000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:42:08.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:08.742+0000 7f6f305de000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:42:09.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[25107]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:09.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:09.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:09.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:09.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:09.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:09.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[29810]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:09.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:09.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:09.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:09.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:09.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:09.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:09.030+0000 7f6f305de000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:42:09.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:09.118+0000 7f6f305de000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:42:09.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:08 smithi173 ceph-mon[25952]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:09.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:09.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:09.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:09.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:09.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:09.852 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:09.520+0000 7f6f305de000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:42:09.852 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:09.650+0000 7f6f305de000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:42:09.852 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:09.741+0000 7f6f305de000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:42:09.852 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:09.853+0000 7f6f305de000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:42:10.060 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:42:10.060 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:42:10.061 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:10.061 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:42:10.061 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:10.061 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:42:10.061 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:09 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:42:10.061 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:09 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:42:10.061 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:09 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:10.061 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:09 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:42:10.061 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:09 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:10.061 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:09 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:42:10.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:09.942+0000 7f6f305de000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:42:10.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:42:10.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:42:10.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:10.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:42:10.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:10.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:09 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:42:10.839 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:10.483+0000 7f6f305de000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:42:11.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[25107]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:11.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[25107]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:11.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[25107]: Standby manager daemon x started 2024-06-21T01:42:11.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:42:11.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:42:11.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:42:11.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:42:11.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[29810]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:11.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[29810]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:11.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[29810]: Standby manager daemon x started 2024-06-21T01:42:11.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:42:11.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:42:11.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:42:11.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:11 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:42:11.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:42:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:42:10.841+0000 7f6f305de000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:42:11.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:11 smithi173 ceph-mon[25952]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:11.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:11 smithi173 ceph-mon[25952]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:11.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:11 smithi173 ceph-mon[25952]: Standby manager daemon x started 2024-06-21T01:42:11.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:11 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:42:11.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:11 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:42:11.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:11 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:42:11.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:11 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2183381162' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:42:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[25107]: mgrmap e14: y(active, since 61s), standbys: x 2024-06-21T01:42:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:42:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3663041329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e18f46e-a84c-4d61-ab85-72135a2a758d"}]: dispatch 2024-06-21T01:42:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3663041329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e18f46e-a84c-4d61-ab85-72135a2a758d"}]': finished 2024-06-21T01:42:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[25107]: osdmap e5: 1 total, 0 up, 1 in 2024-06-21T01:42:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[29810]: mgrmap e14: y(active, since 61s), standbys: x 2024-06-21T01:42:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:42:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3663041329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e18f46e-a84c-4d61-ab85-72135a2a758d"}]: dispatch 2024-06-21T01:42:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3663041329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e18f46e-a84c-4d61-ab85-72135a2a758d"}]': finished 2024-06-21T01:42:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[29810]: osdmap e5: 1 total, 0 up, 1 in 2024-06-21T01:42:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:12 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:12.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:12 smithi173 ceph-mon[25952]: mgrmap e14: y(active, since 61s), standbys: x 2024-06-21T01:42:12.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:12 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:42:12.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:12 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3663041329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e18f46e-a84c-4d61-ab85-72135a2a758d"}]: dispatch 2024-06-21T01:42:12.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:12 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3663041329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e18f46e-a84c-4d61-ab85-72135a2a758d"}]': finished 2024-06-21T01:42:12.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:12 smithi173 ceph-mon[25952]: osdmap e5: 1 total, 0 up, 1 in 2024-06-21T01:42:12.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:12 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:13.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:13 smithi092 ceph-mon[29810]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:13.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:13 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1805120129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:42:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:13 smithi092 ceph-mon[25107]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:13 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1805120129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:42:13.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:13 smithi173 ceph-mon[25952]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:13.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:13 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1805120129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:42:15.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:15 smithi092 ceph-mon[25107]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:15.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:15 smithi092 ceph-mon[29810]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:15.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:15 smithi173 ceph-mon[25952]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:17.493 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:17 smithi092 ceph-mon[25107]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:17.493 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:17 smithi092 ceph-mon[29810]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:17.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:17 smithi173 ceph-mon[25952]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:18.522 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-21T01:42:18.523 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:18.523 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-21T01:42:18.523 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:18.629 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-21T01:42:18.630 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:19.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:19 smithi092 ceph-mon[25107]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:19.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:19 smithi092 ceph-mon[25107]: Deploying daemon osd.0 on smithi092 2024-06-21T01:42:19.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:19 smithi092 ceph-mon[29810]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:19.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:19 smithi092 ceph-mon[29810]: Deploying daemon osd.0 on smithi092 2024-06-21T01:42:19.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:19 smithi173 ceph-mon[25952]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:19.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:19 smithi173 ceph-mon[25952]: Deploying daemon osd.0 on smithi092 2024-06-21T01:42:21.535 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:21 smithi092 ceph-mon[25107]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:21.535 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:21 smithi092 ceph-mon[29810]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:21.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:21 smithi173 ceph-mon[25952]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:23.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:23 smithi173 ceph-mon[25952]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:23.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:23 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:23.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:23 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:23.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:23 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:23.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:23 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:23 smithi092 ceph-mon[25107]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:23 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:23.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:23 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:23.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:23 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:23.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:23 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:23.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:23 smithi092 ceph-mon[29810]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:23.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:23 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:23.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:23 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:23.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:23 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:23.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:23 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:25.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:25 smithi092 ceph-mon[25107]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:25.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:25 smithi092 ceph-mon[29810]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:25.661 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 0 on host 'smithi092' 2024-06-21T01:42:25.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:25 smithi173 ceph-mon[25952]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:26.272 DEBUG:teuthology.orchestra.run.smithi092:osd.0> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.0.service 2024-06-21T01:42:26.316 INFO:tasks.cephadm:Deploying osd.1 on smithi092 with /dev/vg_nvme/lv_3... 2024-06-21T01:42:26.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-21T01:42:26.974 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:26 smithi092 ceph-mon[25107]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:26.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:26 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:26.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:26 smithi092 ceph-mon[25107]: from='osd.0 [v2:172.21.15.92:6802/81657251,v1:172.21.15.92:6803/81657251]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:42:26.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:26 smithi092 ceph-mon[25107]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:42:26.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:26 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:26.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:26 smithi092 ceph-mon[29810]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:26.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:26 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:26.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:26 smithi092 ceph-mon[29810]: from='osd.0 [v2:172.21.15.92:6802/81657251,v1:172.21.15.92:6803/81657251]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:42:26.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:26 smithi092 ceph-mon[29810]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:42:26.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:26 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:26 smithi173 ceph-mon[25952]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:26 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:26 smithi173 ceph-mon[25952]: from='osd.0 [v2:172.21.15.92:6802/81657251,v1:172.21.15.92:6803/81657251]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:42:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:26 smithi173 ceph-mon[25952]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:42:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:26 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:27.898 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-mon[29810]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-21T01:42:27.898 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-mon[29810]: osdmap e6: 1 total, 0 up, 1 in 2024-06-21T01:42:27.898 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:27.898 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-mon[29810]: from='osd.0 [v2:172.21.15.92:6802/81657251,v1:172.21.15.92:6803/81657251]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:42:27.898 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-mon[29810]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:42:27.898 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-mon[25107]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-21T01:42:27.898 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-mon[25107]: osdmap e6: 1 total, 0 up, 1 in 2024-06-21T01:42:27.899 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:27.899 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-mon[25107]: from='osd.0 [v2:172.21.15.92:6802/81657251,v1:172.21.15.92:6803/81657251]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:42:27.899 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-mon[25107]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:42:27.899 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[32647]: 2024-06-21T01:42:27.674+0000 7f10473bb700 -1 osd.0 0 waiting for initial osdmap 2024-06-21T01:42:27.899 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:42:27 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[32647]: 2024-06-21T01:42:27.677+0000 7f1041551700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:42:27.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:27 smithi173 ceph-mon[25952]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-21T01:42:27.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:27 smithi173 ceph-mon[25952]: osdmap e6: 1 total, 0 up, 1 in 2024-06-21T01:42:27.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:27 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:27.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:27 smithi173 ceph-mon[25952]: from='osd.0 [v2:172.21.15.92:6802/81657251,v1:172.21.15.92:6803/81657251]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:42:27.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:27 smithi173 ceph-mon[25952]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:42:28.378 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-21T01:42:28.378 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-21T01:42:28.378 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-06-21T01:42:28.378 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-06-21T01:42:28.378 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124528 s, 842 MB/s 2024-06-21T01:42:28.378 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-06-21T01:42:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:28 smithi173 ceph-mon[25952]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:28.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:28 smithi173 ceph-mon[25952]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:42:28.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:28 smithi173 ceph-mon[25952]: osdmap e7: 1 total, 0 up, 1 in 2024-06-21T01:42:28.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:28 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:28.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:28 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:29.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:28 smithi092 ceph-mon[29810]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:29.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:28 smithi092 ceph-mon[29810]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:42:29.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:28 smithi092 ceph-mon[29810]: osdmap e7: 1 total, 0 up, 1 in 2024-06-21T01:42:29.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:28 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:29.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:28 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:29.064 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:28 smithi092 ceph-mon[25107]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:29.064 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:28 smithi092 ceph-mon[25107]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:42:29.064 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:28 smithi092 ceph-mon[25107]: osdmap e7: 1 total, 0 up, 1 in 2024-06-21T01:42:29.064 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:28 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:29.064 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:28 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:29.911 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:29 smithi092 ceph-mon[25107]: purged_snaps scrub starts 2024-06-21T01:42:29.912 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:29 smithi092 ceph-mon[25107]: purged_snaps scrub ok 2024-06-21T01:42:29.912 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:29 smithi092 ceph-mon[25107]: osd.0 [v2:172.21.15.92:6802/81657251,v1:172.21.15.92:6803/81657251] boot 2024-06-21T01:42:29.912 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:29 smithi092 ceph-mon[25107]: osdmap e8: 1 total, 1 up, 1 in 2024-06-21T01:42:29.912 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:29 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:29.912 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:29 smithi092 ceph-mon[29810]: purged_snaps scrub starts 2024-06-21T01:42:29.912 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:29 smithi092 ceph-mon[29810]: purged_snaps scrub ok 2024-06-21T01:42:29.912 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:29 smithi092 ceph-mon[29810]: osd.0 [v2:172.21.15.92:6802/81657251,v1:172.21.15.92:6803/81657251] boot 2024-06-21T01:42:29.912 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:29 smithi092 ceph-mon[29810]: osdmap e8: 1 total, 1 up, 1 in 2024-06-21T01:42:29.912 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:29 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:29.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:29 smithi173 ceph-mon[25952]: purged_snaps scrub starts 2024-06-21T01:42:29.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:29 smithi173 ceph-mon[25952]: purged_snaps scrub ok 2024-06-21T01:42:29.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:29 smithi173 ceph-mon[25952]: osd.0 [v2:172.21.15.92:6802/81657251,v1:172.21.15.92:6803/81657251] boot 2024-06-21T01:42:29.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:29 smithi173 ceph-mon[25952]: osdmap e8: 1 total, 1 up, 1 in 2024-06-21T01:42:29.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:29 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:42:30.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_3 2024-06-21T01:42:30.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:30 smithi092 ceph-mon[29810]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:30.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:30 smithi092 ceph-mon[29810]: osdmap e9: 1 total, 1 up, 1 in 2024-06-21T01:42:30.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:30 smithi092 ceph-mon[25107]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:30.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:30 smithi092 ceph-mon[25107]: osdmap e9: 1 total, 1 up, 1 in 2024-06-21T01:42:30.951 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:30 smithi173 ceph-mon[25952]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-21T01:42:30.951 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:30 smithi173 ceph-mon[25952]: osdmap e9: 1 total, 1 up, 1 in 2024-06-21T01:42:32.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[25107]: Detected new or changed devices on smithi092 2024-06-21T01:42:32.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:32.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:42:32.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[25107]: Adjusting osd_memory_target on smithi092 to 16064M 2024-06-21T01:42:32.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:32.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:32.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:32.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[29810]: Detected new or changed devices on smithi092 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[29810]: Adjusting osd_memory_target on smithi092 to 16064M 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:32.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:31 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:32.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:31 smithi173 ceph-mon[25952]: Detected new or changed devices on smithi092 2024-06-21T01:42:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:31 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:31 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:42:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:31 smithi173 ceph-mon[25952]: Adjusting osd_memory_target on smithi092 to 16064M 2024-06-21T01:42:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:31 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:31 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:31 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:31 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:31 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:31 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:33 smithi092 ceph-mon[25107]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:33 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:42:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:33 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:42:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:33 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:33.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:33 smithi092 ceph-mon[29810]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:33.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:33 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:42:33.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:33 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:42:33.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:33 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:33.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:33 smithi173 ceph-mon[25952]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:33.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:33 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:42:33.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:33 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:42:33.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:33 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:34.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:34 smithi173 ceph-mon[25952]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:34.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:34 smithi092 ceph-mon[25107]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:34.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:34 smithi092 ceph-mon[29810]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:35.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:35 smithi173 ceph-mon[25952]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:35.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:35 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1493750212' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06f0cb8c-8380-46d1-bf8e-cddd9930f5c1"}]: dispatch 2024-06-21T01:42:35.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:35 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06f0cb8c-8380-46d1-bf8e-cddd9930f5c1"}]: dispatch 2024-06-21T01:42:35.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:35 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06f0cb8c-8380-46d1-bf8e-cddd9930f5c1"}]': finished 2024-06-21T01:42:35.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:35 smithi173 ceph-mon[25952]: osdmap e10: 2 total, 1 up, 2 in 2024-06-21T01:42:35.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:35 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:35.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[25107]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:35.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1493750212' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06f0cb8c-8380-46d1-bf8e-cddd9930f5c1"}]: dispatch 2024-06-21T01:42:35.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06f0cb8c-8380-46d1-bf8e-cddd9930f5c1"}]: dispatch 2024-06-21T01:42:35.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06f0cb8c-8380-46d1-bf8e-cddd9930f5c1"}]': finished 2024-06-21T01:42:35.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[25107]: osdmap e10: 2 total, 1 up, 2 in 2024-06-21T01:42:35.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:35.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[29810]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:35.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1493750212' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06f0cb8c-8380-46d1-bf8e-cddd9930f5c1"}]: dispatch 2024-06-21T01:42:35.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06f0cb8c-8380-46d1-bf8e-cddd9930f5c1"}]: dispatch 2024-06-21T01:42:35.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06f0cb8c-8380-46d1-bf8e-cddd9930f5c1"}]': finished 2024-06-21T01:42:35.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[29810]: osdmap e10: 2 total, 1 up, 2 in 2024-06-21T01:42:35.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:35 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:36.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:36 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/723340212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:42:36.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:36 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/723340212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:42:36.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:36 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/723340212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:42:37.697 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:37 smithi092 ceph-mon[25107]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:37.697 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:37 smithi092 ceph-mon[29810]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:37.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:37 smithi173 ceph-mon[25952]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:39.706 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:39 smithi092 ceph-mon[25107]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:39.707 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:39 smithi092 ceph-mon[29810]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:39.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:39 smithi173 ceph-mon[25952]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:41.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:41 smithi092 ceph-mon[29810]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:41.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:41 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-21T01:42:41.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:41 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:41.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:41 smithi092 ceph-mon[25107]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:41.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:41 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-21T01:42:41.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:41 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:41.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:41 smithi173 ceph-mon[25952]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:41.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:41 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-21T01:42:41.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:41 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:42.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:42 smithi092 ceph-mon[25107]: Deploying daemon osd.1 on smithi092 2024-06-21T01:42:42.598 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:42 smithi092 ceph-mon[29810]: Deploying daemon osd.1 on smithi092 2024-06-21T01:42:42.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:42 smithi173 ceph-mon[25952]: Deploying daemon osd.1 on smithi092 2024-06-21T01:42:43.661 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:43 smithi092 ceph-mon[25107]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:43.661 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:43 smithi092 ceph-mon[29810]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:43.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:43 smithi173 ceph-mon[25952]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:45.695 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:45 smithi092 ceph-mon[25107]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:45.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:45 smithi092 ceph-mon[29810]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:45.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:45 smithi173 ceph-mon[25952]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:46.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:46 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:46.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:46 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:46.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:46 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:46.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:46 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:46.480 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:46 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:46.480 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:46 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:46.480 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:46 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:46.480 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:46 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:46.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:46 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:46.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:46 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:46.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:46 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:46.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:46 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:47.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:47 smithi092 ceph-mon[25107]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:47.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:47 smithi092 ceph-mon[29810]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:47.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:47 smithi173 ceph-mon[25952]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:49.192 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 1 on host 'smithi092' 2024-06-21T01:42:49.550 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[29810]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:49.550 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:49.550 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:49.550 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[29810]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-21T01:42:49.550 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:49.550 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:49.550 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:49.550 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:49.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[25107]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:49.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:49.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:49.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[25107]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-21T01:42:49.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:49.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:49.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:49.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:49.552 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:49 smithi173 ceph-mon[25952]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:49.552 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:49.552 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:49.552 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:49 smithi173 ceph-mon[25952]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-21T01:42:49.552 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:49.552 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:42:49.552 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:49.552 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:42:49.817 DEBUG:teuthology.orchestra.run.smithi092:osd.1> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.1.service 2024-06-21T01:42:49.820 INFO:tasks.cephadm:Deploying osd.2 on smithi092 with /dev/vg_nvme/lv_2... 2024-06-21T01:42:49.820 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-21T01:42:50.682 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:50 smithi173 ceph-mon[25952]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-21T01:42:50.682 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:50 smithi173 ceph-mon[25952]: osdmap e11: 2 total, 1 up, 2 in 2024-06-21T01:42:50.682 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:50 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:50.682 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:50 smithi173 ceph-mon[25952]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:42:50.682 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:50 smithi173 ceph-mon[25952]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:50.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-mon[29810]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-21T01:42:50.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-mon[29810]: osdmap e11: 2 total, 1 up, 2 in 2024-06-21T01:42:50.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:50.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-mon[29810]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:42:50.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-mon[29810]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:50.695 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-mon[25107]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-21T01:42:50.695 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-mon[25107]: osdmap e11: 2 total, 1 up, 2 in 2024-06-21T01:42:50.696 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:50.696 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-mon[25107]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:42:50.696 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-mon[25107]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:50.696 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[35593]: 2024-06-21T01:42:50.428+0000 7f0219d01700 -1 osd.1 0 waiting for initial osdmap 2024-06-21T01:42:50.696 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:42:50 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[35593]: 2024-06-21T01:42:50.431+0000 7f0213e97700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:42:51.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:51 smithi173 ceph-mon[25952]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:42:51.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:51 smithi173 ceph-mon[25952]: osdmap e12: 2 total, 1 up, 2 in 2024-06-21T01:42:51.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:51 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:51.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:51 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:51.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:51 smithi092 ceph-mon[25107]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:42:51.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:51 smithi092 ceph-mon[25107]: osdmap e12: 2 total, 1 up, 2 in 2024-06-21T01:42:51.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:51 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:51.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:51 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:51.756 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:51 smithi092 ceph-mon[29810]: from='osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:42:51.756 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:51 smithi092 ceph-mon[29810]: osdmap e12: 2 total, 1 up, 2 in 2024-06-21T01:42:51.756 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:51 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:51.756 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:51 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:52.686 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[29810]: purged_snaps scrub starts 2024-06-21T01:42:52.687 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[29810]: purged_snaps scrub ok 2024-06-21T01:42:52.687 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[29810]: osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642] boot 2024-06-21T01:42:52.687 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[29810]: osdmap e13: 2 total, 2 up, 2 in 2024-06-21T01:42:52.687 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:52.687 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[29810]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:52.687 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[25107]: purged_snaps scrub starts 2024-06-21T01:42:52.687 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[25107]: purged_snaps scrub ok 2024-06-21T01:42:52.687 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[25107]: osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642] boot 2024-06-21T01:42:52.687 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[25107]: osdmap e13: 2 total, 2 up, 2 in 2024-06-21T01:42:52.688 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:52.688 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:52 smithi092 ceph-mon[25107]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:52.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:52 smithi173 ceph-mon[25952]: purged_snaps scrub starts 2024-06-21T01:42:52.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:52 smithi173 ceph-mon[25952]: purged_snaps scrub ok 2024-06-21T01:42:52.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:52 smithi173 ceph-mon[25952]: osd.1 [v2:172.21.15.92:6810/185612642,v1:172.21.15.92:6811/185612642] boot 2024-06-21T01:42:52.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:52 smithi173 ceph-mon[25952]: osdmap e13: 2 total, 2 up, 2 in 2024-06-21T01:42:52.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:52 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:42:52.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:52 smithi173 ceph-mon[25952]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-21T01:42:53.356 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-21T01:42:53.357 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-21T01:42:53.357 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-06-21T01:42:53.357 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-06-21T01:42:53.357 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123886 s, 846 MB/s 2024-06-21T01:42:53.357 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-06-21T01:42:53.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:53 smithi173 ceph-mon[25952]: osdmap e14: 2 total, 2 up, 2 in 2024-06-21T01:42:53.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:53 smithi092 ceph-mon[29810]: osdmap e14: 2 total, 2 up, 2 in 2024-06-21T01:42:53.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:53 smithi092 ceph-mon[25107]: osdmap e14: 2 total, 2 up, 2 in 2024-06-21T01:42:54.251 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_2 2024-06-21T01:42:54.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[29810]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:42:54.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[29810]: Detected new or changed devices on smithi092 2024-06-21T01:42:54.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:54.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[29810]: Adjusting osd_memory_target on smithi092 to 8032M 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[25107]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[25107]: Detected new or changed devices on smithi092 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[25107]: Adjusting osd_memory_target on smithi092 to 8032M 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:54.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:54.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:54 smithi173 ceph-mon[25952]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:42:54.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:54 smithi173 ceph-mon[25952]: Detected new or changed devices on smithi092 2024-06-21T01:42:54.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:54 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:54.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:54 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:42:54.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:54 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:42:54.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:54 smithi173 ceph-mon[25952]: Adjusting osd_memory_target on smithi092 to 8032M 2024-06-21T01:42:54.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:54 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:54.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:54 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:42:56.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:56 smithi092 ceph-mon[25107]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:42:56.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:56 smithi092 ceph-mon[25107]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:56.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:56 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:42:56.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:56 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:42:56.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:56 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:56.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:56 smithi092 ceph-mon[29810]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:42:56.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:56 smithi092 ceph-mon[29810]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:56.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:56 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:42:56.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:56 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:42:56.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:56 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:56.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:56 smithi173 ceph-mon[25952]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:42:56.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:56 smithi173 ceph-mon[25952]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:42:56.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:56 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:42:56.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:56 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:42:56.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:56 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:42:58.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[29810]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:42:58.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/4066111269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51c9d336-d57f-449e-bbda-07d212c4224d"}]: dispatch 2024-06-21T01:42:58.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51c9d336-d57f-449e-bbda-07d212c4224d"}]: dispatch 2024-06-21T01:42:58.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51c9d336-d57f-449e-bbda-07d212c4224d"}]': finished 2024-06-21T01:42:58.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[29810]: osdmap e15: 3 total, 2 up, 3 in 2024-06-21T01:42:58.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:42:58.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[25107]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:42:58.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/4066111269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51c9d336-d57f-449e-bbda-07d212c4224d"}]: dispatch 2024-06-21T01:42:58.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51c9d336-d57f-449e-bbda-07d212c4224d"}]: dispatch 2024-06-21T01:42:58.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51c9d336-d57f-449e-bbda-07d212c4224d"}]': finished 2024-06-21T01:42:58.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[25107]: osdmap e15: 3 total, 2 up, 3 in 2024-06-21T01:42:58.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:58 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:42:58.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:58 smithi173 ceph-mon[25952]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:42:58.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:58 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/4066111269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51c9d336-d57f-449e-bbda-07d212c4224d"}]: dispatch 2024-06-21T01:42:58.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:58 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51c9d336-d57f-449e-bbda-07d212c4224d"}]: dispatch 2024-06-21T01:42:58.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:58 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51c9d336-d57f-449e-bbda-07d212c4224d"}]': finished 2024-06-21T01:42:58.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:58 smithi173 ceph-mon[25952]: osdmap e15: 3 total, 2 up, 3 in 2024-06-21T01:42:58.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:58 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:42:59.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:42:59 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2576663299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:42:59.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:42:59 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2576663299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:42:59.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:42:59 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2576663299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:43:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:00 smithi092 ceph-mon[25107]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:00.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:00 smithi092 ceph-mon[29810]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:00.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:00 smithi173 ceph-mon[25952]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:03.524 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:03 smithi092 ceph-mon[25107]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:03.524 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:03 smithi092 ceph-mon[29810]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:03.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:03 smithi173 ceph-mon[25952]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:05.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:05 smithi173 ceph-mon[25952]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:05.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-21T01:43:05.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:05.809 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:05 smithi092 ceph-mon[25107]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:05.810 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-21T01:43:05.810 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:05.810 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:05 smithi092 ceph-mon[29810]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:05.810 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-21T01:43:05.810 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:06.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:06 smithi092 ceph-mon[25107]: Deploying daemon osd.2 on smithi092 2024-06-21T01:43:06.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:06 smithi092 ceph-mon[29810]: Deploying daemon osd.2 on smithi092 2024-06-21T01:43:06.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:06 smithi173 ceph-mon[25952]: Deploying daemon osd.2 on smithi092 2024-06-21T01:43:07.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:07 smithi092 ceph-mon[25107]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:07.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:07 smithi092 ceph-mon[29810]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:07.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:07 smithi173 ceph-mon[25952]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:09.684 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:09 smithi092 ceph-mon[25107]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:09.685 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:09 smithi092 ceph-mon[29810]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:09.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:09 smithi173 ceph-mon[25952]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:10.682 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:43:10.682 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:43:10.682 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:10.682 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:10.682 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:10.683 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:10.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:43:10.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:43:10.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:10.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:10.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:10.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:10.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:10 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:43:10.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:10 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:43:10.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:10 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:10.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:10 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:10.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:10 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:10.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:10 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:11.525 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:11 smithi092 ceph-mon[29810]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:11.526 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:11 smithi092 ceph-mon[25107]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:11.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:11 smithi173 ceph-mon[25952]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:12.834 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 2 on host 'smithi092' 2024-06-21T01:43:13.472 DEBUG:teuthology.orchestra.run.smithi092:osd.2> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.2.service 2024-06-21T01:43:13.475 INFO:tasks.cephadm:Deploying osd.3 on smithi092 with /dev/vg_nvme/lv_1... 2024-06-21T01:43:13.476 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-21T01:43:13.671 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:13 smithi092 ceph-mon[25107]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:13.671 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:13 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:13.672 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:13 smithi092 ceph-mon[25107]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-21T01:43:13.672 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:13 smithi092 ceph-mon[29810]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:13.672 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:13 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:13.672 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:13 smithi092 ceph-mon[29810]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-21T01:43:13.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:13 smithi173 ceph-mon[25952]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:13.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:13 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:13.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:13 smithi173 ceph-mon[25952]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-21T01:43:14.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[25107]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:14.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:14.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[25107]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-21T01:43:14.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[25107]: osdmap e16: 3 total, 2 up, 3 in 2024-06-21T01:43:14.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:14.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[25107]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:43:14.840 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[29810]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:14.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:14.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[29810]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-21T01:43:14.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[29810]: osdmap e16: 3 total, 2 up, 3 in 2024-06-21T01:43:14.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:14.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-mon[29810]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:43:14.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:14 smithi173 ceph-mon[25952]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:14.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:14 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:14.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:14 smithi173 ceph-mon[25952]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-21T01:43:14.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:14 smithi173 ceph-mon[25952]: osdmap e16: 3 total, 2 up, 3 in 2024-06-21T01:43:14.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:14 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:14.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:14 smithi173 ceph-mon[25952]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:43:15.101 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[38495]: 2024-06-21T01:43:14.842+0000 7f58fa4ba700 -1 osd.2 0 waiting for initial osdmap 2024-06-21T01:43:15.102 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:43:14 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[38495]: 2024-06-21T01:43:14.845+0000 7f58f3e4f700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:43:16.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:15 smithi092 ceph-mon[25107]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:43:16.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:15 smithi092 ceph-mon[25107]: osdmap e17: 3 total, 2 up, 3 in 2024-06-21T01:43:16.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:15 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:16.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:15 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:16.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:15 smithi092 ceph-mon[29810]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:43:16.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:15 smithi092 ceph-mon[29810]: osdmap e17: 3 total, 2 up, 3 in 2024-06-21T01:43:16.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:15 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:16.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:15 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:15 smithi173 ceph-mon[25952]: from='osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:43:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:15 smithi173 ceph-mon[25952]: osdmap e17: 3 total, 2 up, 3 in 2024-06-21T01:43:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:15 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:15 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:16.867 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-21T01:43:16.867 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-21T01:43:16.867 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-06-21T01:43:16.867 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-06-21T01:43:16.867 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012569 s, 834 MB/s 2024-06-21T01:43:16.868 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-06-21T01:43:17.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[25107]: purged_snaps scrub starts 2024-06-21T01:43:17.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[25107]: purged_snaps scrub ok 2024-06-21T01:43:17.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[25107]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:17.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[25107]: osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716] boot 2024-06-21T01:43:17.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[25107]: osdmap e18: 3 total, 3 up, 3 in 2024-06-21T01:43:17.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:17.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[29810]: purged_snaps scrub starts 2024-06-21T01:43:17.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[29810]: purged_snaps scrub ok 2024-06-21T01:43:17.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[29810]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:17.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[29810]: osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716] boot 2024-06-21T01:43:17.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[29810]: osdmap e18: 3 total, 3 up, 3 in 2024-06-21T01:43:17.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:16 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:17.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:16 smithi173 ceph-mon[25952]: purged_snaps scrub starts 2024-06-21T01:43:17.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:16 smithi173 ceph-mon[25952]: purged_snaps scrub ok 2024-06-21T01:43:17.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:16 smithi173 ceph-mon[25952]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-21T01:43:17.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:16 smithi173 ceph-mon[25952]: osd.2 [v2:172.21.15.92:6818/2600120716,v1:172.21.15.92:6819/2600120716] boot 2024-06-21T01:43:17.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:16 smithi173 ceph-mon[25952]: osdmap e18: 3 total, 3 up, 3 in 2024-06-21T01:43:17.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:16 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:43:17.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_1 2024-06-21T01:43:18.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:17 smithi092 ceph-mon[25107]: osdmap e19: 3 total, 3 up, 3 in 2024-06-21T01:43:18.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:17 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-21T01:43:18.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:17 smithi092 ceph-mon[29810]: osdmap e19: 3 total, 3 up, 3 in 2024-06-21T01:43:18.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:17 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-21T01:43:18.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:17 smithi173 ceph-mon[25952]: osdmap e19: 3 total, 3 up, 3 in 2024-06-21T01:43:18.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:17 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-21T01:43:19.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:19.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-21T01:43:19.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: osdmap e20: 3 total, 3 up, 3 in 2024-06-21T01:43:19.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-21T01:43:19.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: Detected new or changed devices on smithi092 2024-06-21T01:43:19.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:19.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:19.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:19.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: Adjusting osd_memory_target on smithi092 to 5354M 2024-06-21T01:43:19.130 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.131 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.131 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:19.131 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:19.131 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:19.131 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:18 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.153 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: osdmap e20: 3 total, 3 up, 3 in 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: Detected new or changed devices on smithi092 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: Adjusting osd_memory_target on smithi092 to 5354M 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:19.154 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:19.155 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:19.155 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.155 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:19.155 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-21T01:43:19.155 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: osdmap e20: 3 total, 3 up, 3 in 2024-06-21T01:43:19.155 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-21T01:43:19.155 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: Detected new or changed devices on smithi092 2024-06-21T01:43:19.155 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.155 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:19.155 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:19.155 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:19.156 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: Adjusting osd_memory_target on smithi092 to 5354M 2024-06-21T01:43:19.156 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.156 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.156 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:19.156 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:19.156 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:19.156 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:18 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.431 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39780]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-21T01:43:19.431 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39780]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:19.431 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39780]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:19.720 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39806]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-21T01:43:19.720 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39806]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:19.720 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39806]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:19.983 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:19 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-21T01:43:19.983 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:19 smithi092 ceph-mon[25107]: osdmap e21: 3 total, 3 up, 3 in 2024-06-21T01:43:19.983 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:19 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.983 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:19 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-21T01:43:19.983 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:19 smithi092 ceph-mon[29810]: osdmap e21: 3 total, 3 up, 3 in 2024-06-21T01:43:19.984 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:19 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:19.984 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39831]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-21T01:43:19.984 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39831]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:19.984 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39831]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:20.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:19 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-21T01:43:20.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:19 smithi173 ceph-mon[25952]: osdmap e21: 3 total, 3 up, 3 in 2024-06-21T01:43:20.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:19 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:20.234 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39871]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-21T01:43:20.234 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39871]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:20.235 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:43:19 smithi092 sudo[39871]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:20.493 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39915]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-21T01:43:20.493 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39915]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:20.493 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39915]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:20.751 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39941]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-21T01:43:20.751 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39941]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:20.751 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39941]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:20.751 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39937]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-21T01:43:20.752 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39937]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:20.752 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39937]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:21.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39946]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-21T01:43:21.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39946]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:21.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:20 smithi092 sudo[39946]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:21.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[25107]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:21.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[25107]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[25107]: osdmap e22: 3 total, 3 up, 3 in 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[25107]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[25107]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[29810]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[29810]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[29810]: osdmap e22: 3 total, 3 up, 3 in 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[29810]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:43:21.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:43:21.038 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:21.038 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 ceph-mon[29810]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-21T01:43:21.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 sudo[39968]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-21T01:43:21.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 sudo[39968]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:21.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 sudo[39968]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:21.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:21 smithi173 ceph-mon[25952]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:21.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:21 smithi173 ceph-mon[25952]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:43:21.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:21 smithi173 ceph-mon[25952]: osdmap e22: 3 total, 3 up, 3 in 2024-06-21T01:43:21.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:21 smithi173 ceph-mon[25952]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-21T01:43:21.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:21 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:43:21.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:21 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:43:21.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:21 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:21.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:21 smithi173 ceph-mon[25952]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-21T01:43:21.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 sudo[39978]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-21T01:43:21.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 sudo[39978]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:21.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:21 smithi092 sudo[39978]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:22.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:21 smithi173 sudo[27469]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-21T01:43:22.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:21 smithi173 sudo[27469]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:22.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:21 smithi173 sudo[27469]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:22.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-21T01:43:22.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:43:22.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:43:22.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:43:22.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-21T01:43:22.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:43:22.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:43:22.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:43:22.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-21T01:43:22.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:43:22.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:43:22.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:43:22.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:43:22.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 sudo[27472]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-21T01:43:22.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 sudo[27472]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 sudo[27472]: pam_unix(sudo:session): session closed for user root 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:43:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:43:23.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[25107]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:23.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[25107]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-21T01:43:23.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[25107]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-21T01:43:23.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:23.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1475262090' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22670eae-7380-47c6-ab26-9e9382026568"}]: dispatch 2024-06-21T01:43:23.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1475262090' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22670eae-7380-47c6-ab26-9e9382026568"}]': finished 2024-06-21T01:43:23.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[25107]: osdmap e23: 4 total, 3 up, 4 in 2024-06-21T01:43:23.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:23.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/4203993890' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:43:23.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[29810]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:23.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[29810]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-21T01:43:23.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[29810]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-21T01:43:23.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:23.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1475262090' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22670eae-7380-47c6-ab26-9e9382026568"}]: dispatch 2024-06-21T01:43:23.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1475262090' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22670eae-7380-47c6-ab26-9e9382026568"}]': finished 2024-06-21T01:43:23.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[29810]: osdmap e23: 4 total, 3 up, 4 in 2024-06-21T01:43:23.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:23.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:23 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/4203993890' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:43:23.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:23 smithi173 ceph-mon[25952]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:23.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:23 smithi173 ceph-mon[25952]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-21T01:43:23.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:23 smithi173 ceph-mon[25952]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-21T01:43:23.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:23 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:23.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:23 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1475262090' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22670eae-7380-47c6-ab26-9e9382026568"}]: dispatch 2024-06-21T01:43:23.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:23 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1475262090' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22670eae-7380-47c6-ab26-9e9382026568"}]': finished 2024-06-21T01:43:23.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:23 smithi173 ceph-mon[25952]: osdmap e23: 4 total, 3 up, 4 in 2024-06-21T01:43:23.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:23 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:23.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:23 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/4203993890' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:43:25.236 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:25 smithi092 ceph-mon[25107]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:25.236 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:25 smithi092 ceph-mon[29810]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:25.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:25 smithi173 ceph-mon[25952]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:27.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:27 smithi173 ceph-mon[25952]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:27.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:27 smithi092 ceph-mon[25107]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:27.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:27 smithi092 ceph-mon[29810]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:28.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:28 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-21T01:43:28.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:28 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:28.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:28 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-21T01:43:28.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:28 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:28.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:28 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-21T01:43:28.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:28 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:29.686 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:29 smithi092 ceph-mon[25107]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:29.687 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:29 smithi092 ceph-mon[25107]: Deploying daemon osd.3 on smithi092 2024-06-21T01:43:29.687 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:29 smithi092 ceph-mon[29810]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:29.687 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:29 smithi092 ceph-mon[29810]: Deploying daemon osd.3 on smithi092 2024-06-21T01:43:29.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:29 smithi173 ceph-mon[25952]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:29.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:29 smithi173 ceph-mon[25952]: Deploying daemon osd.3 on smithi092 2024-06-21T01:43:31.682 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:31 smithi092 ceph-mon[25107]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:31.683 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:31 smithi092 ceph-mon[29810]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:31.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:31 smithi173 ceph-mon[25952]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:33.570 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:33 smithi173 ceph-mon[25952]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:33.570 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:33 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:33.570 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:33 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:33.570 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:33 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:33.570 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:33 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:33 smithi092 ceph-mon[25107]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:33 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:33 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:33 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:33.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:33 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:33.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:33 smithi092 ceph-mon[29810]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:33.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:33 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:33.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:33 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:33.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:33 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:33.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:33 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:35.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:35 smithi092 ceph-mon[25107]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:35.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:35 smithi092 ceph-mon[29810]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:35.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:35 smithi173 ceph-mon[25952]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:36.386 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 3 on host 'smithi092' 2024-06-21T01:43:36.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:36 smithi092 ceph-mon[25107]: from='osd.3 [v2:172.21.15.92:6826/2261799524,v1:172.21.15.92:6827/2261799524]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:43:36.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:36 smithi092 ceph-mon[25107]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:43:36.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:36 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:36.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:36 smithi092 ceph-mon[29810]: from='osd.3 [v2:172.21.15.92:6826/2261799524,v1:172.21.15.92:6827/2261799524]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:43:36.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:36 smithi092 ceph-mon[29810]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:43:36.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:36 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:36.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:36 smithi173 ceph-mon[25952]: from='osd.3 [v2:172.21.15.92:6826/2261799524,v1:172.21.15.92:6827/2261799524]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:43:36.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:36 smithi173 ceph-mon[25952]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:43:36.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:36 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:36.899 DEBUG:teuthology.orchestra.run.smithi092:osd.3> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.3.service 2024-06-21T01:43:36.901 INFO:tasks.cephadm:Deploying osd.4 on smithi173 with /dev/vg_nvme/lv_4... 2024-06-21T01:43:36.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-21T01:43:37.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:37 smithi173 ceph-mon[25952]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:37.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:37 smithi173 ceph-mon[25952]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-21T01:43:37.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:37 smithi173 ceph-mon[25952]: osdmap e24: 4 total, 3 up, 4 in 2024-06-21T01:43:37.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:37 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:37.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:37 smithi173 ceph-mon[25952]: from='osd.3 [v2:172.21.15.92:6826/2261799524,v1:172.21.15.92:6827/2261799524]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:43:37.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:37 smithi173 ceph-mon[25952]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:43:37.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:37 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:37.739 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[25107]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:37.739 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[25107]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-21T01:43:37.739 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[25107]: osdmap e24: 4 total, 3 up, 4 in 2024-06-21T01:43:37.739 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:37.739 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[25107]: from='osd.3 [v2:172.21.15.92:6826/2261799524,v1:172.21.15.92:6827/2261799524]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:43:37.739 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[25107]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:43:37.739 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:37.740 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[29810]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:37.740 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[29810]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-21T01:43:37.740 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[29810]: osdmap e24: 4 total, 3 up, 4 in 2024-06-21T01:43:37.740 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:37.740 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[29810]: from='osd.3 [v2:172.21.15.92:6826/2261799524,v1:172.21.15.92:6827/2261799524]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:43:37.740 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[29810]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:43:37.740 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:37.740 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[41451]: 2024-06-21T01:43:37.421+0000 7f0eded14700 -1 osd.3 0 waiting for initial osdmap 2024-06-21T01:43:37.741 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:43:37 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[41451]: 2024-06-21T01:43:37.424+0000 7f0ed7ea8700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:43:38.699 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:38 smithi092 ceph-mon[25107]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:43:38.700 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:38 smithi092 ceph-mon[25107]: osdmap e25: 4 total, 3 up, 4 in 2024-06-21T01:43:38.700 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:38 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:38.700 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:38 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:38.700 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:38 smithi092 ceph-mon[29810]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:43:38.700 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:38 smithi092 ceph-mon[29810]: osdmap e25: 4 total, 3 up, 4 in 2024-06-21T01:43:38.700 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:38 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:38.700 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:38 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:38.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:38 smithi173 ceph-mon[25952]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-06-21T01:43:38.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:38 smithi173 ceph-mon[25952]: osdmap e25: 4 total, 3 up, 4 in 2024-06-21T01:43:38.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:38 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:38.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:38 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:38.745 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-21T01:43:38.746 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-21T01:43:38.746 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-06-21T01:43:38.746 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-06-21T01:43:38.746 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112107 s, 935 MB/s 2024-06-21T01:43:38.746 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-06-21T01:43:39.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-06-21T01:43:39.600 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:39 smithi173 ceph-mon[25952]: purged_snaps scrub starts 2024-06-21T01:43:39.600 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:39 smithi173 ceph-mon[25952]: purged_snaps scrub ok 2024-06-21T01:43:39.600 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:39 smithi173 ceph-mon[25952]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:39.600 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:39 smithi173 ceph-mon[25952]: osd.3 [v2:172.21.15.92:6826/2261799524,v1:172.21.15.92:6827/2261799524] boot 2024-06-21T01:43:39.600 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:39 smithi173 ceph-mon[25952]: osdmap e26: 4 total, 4 up, 4 in 2024-06-21T01:43:39.600 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:39 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:39.760 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[25107]: purged_snaps scrub starts 2024-06-21T01:43:39.760 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[25107]: purged_snaps scrub ok 2024-06-21T01:43:39.760 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[25107]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:39.760 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[25107]: osd.3 [v2:172.21.15.92:6826/2261799524,v1:172.21.15.92:6827/2261799524] boot 2024-06-21T01:43:39.760 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[25107]: osdmap e26: 4 total, 4 up, 4 in 2024-06-21T01:43:39.760 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:39.761 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[29810]: purged_snaps scrub starts 2024-06-21T01:43:39.761 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[29810]: purged_snaps scrub ok 2024-06-21T01:43:39.761 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[29810]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-21T01:43:39.761 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[29810]: osd.3 [v2:172.21.15.92:6826/2261799524,v1:172.21.15.92:6827/2261799524] boot 2024-06-21T01:43:39.761 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[29810]: osdmap e26: 4 total, 4 up, 4 in 2024-06-21T01:43:39.761 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:39 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: osdmap e27: 4 total, 4 up, 4 in 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: Detected new or changed devices on smithi092 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: Adjusting osd_memory_target on smithi092 to 4016M 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:40.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:40.500 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:40.500 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:40.500 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:40 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:40.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: osdmap e27: 4 total, 4 up, 4 in 2024-06-21T01:43:40.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: Detected new or changed devices on smithi092 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: Adjusting osd_memory_target on smithi092 to 4016M 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:40.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: osdmap e27: 4 total, 4 up, 4 in 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: Detected new or changed devices on smithi092 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: Adjusting osd_memory_target on smithi092 to 4016M 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:40.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:40.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:40.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:40.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:40 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:41.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:41 smithi173 ceph-mon[25952]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:43:41.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:41 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:43:41.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:41 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:43:41.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:41 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:41.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:41 smithi092 ceph-mon[25107]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:43:41.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:41 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:43:41.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:41 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:43:41.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:41 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:41.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:41 smithi092 ceph-mon[29810]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:43:41.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:41 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:43:41.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:41 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:43:41.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:41 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:42.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:42 smithi173 ceph-mon[25952]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:42.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:42 smithi092 ceph-mon[25107]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:42.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:42 smithi092 ceph-mon[29810]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:43.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:43 smithi173 ceph-mon[25952]: from='client.? 172.21.15.173:0/2519946906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6c0946-0039-494e-b7fb-92bbdfb459c3"}]: dispatch 2024-06-21T01:43:43.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:43 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6c0946-0039-494e-b7fb-92bbdfb459c3"}]: dispatch 2024-06-21T01:43:43.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:43 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f6c0946-0039-494e-b7fb-92bbdfb459c3"}]': finished 2024-06-21T01:43:43.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:43 smithi173 ceph-mon[25952]: osdmap e28: 5 total, 4 up, 5 in 2024-06-21T01:43:43.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:43 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:43:43.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:43 smithi092 ceph-mon[25107]: from='client.? 172.21.15.173:0/2519946906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6c0946-0039-494e-b7fb-92bbdfb459c3"}]: dispatch 2024-06-21T01:43:43.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:43 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6c0946-0039-494e-b7fb-92bbdfb459c3"}]: dispatch 2024-06-21T01:43:43.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:43 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f6c0946-0039-494e-b7fb-92bbdfb459c3"}]': finished 2024-06-21T01:43:43.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:43 smithi092 ceph-mon[25107]: osdmap e28: 5 total, 4 up, 5 in 2024-06-21T01:43:43.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:43 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:43:43.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:43 smithi092 ceph-mon[29810]: from='client.? 172.21.15.173:0/2519946906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6c0946-0039-494e-b7fb-92bbdfb459c3"}]: dispatch 2024-06-21T01:43:43.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:43 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f6c0946-0039-494e-b7fb-92bbdfb459c3"}]: dispatch 2024-06-21T01:43:43.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:43 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f6c0946-0039-494e-b7fb-92bbdfb459c3"}]': finished 2024-06-21T01:43:43.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:43 smithi092 ceph-mon[29810]: osdmap e28: 5 total, 4 up, 5 in 2024-06-21T01:43:43.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:43 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:43:44.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:44 smithi173 ceph-mon[25952]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:44.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:44 smithi173 ceph-mon[25952]: from='client.? 172.21.15.173:0/831357283' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:43:44.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:44 smithi092 ceph-mon[25107]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:44.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:44 smithi092 ceph-mon[25107]: from='client.? 172.21.15.173:0/831357283' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:43:44.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:44 smithi092 ceph-mon[29810]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:44.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:44 smithi092 ceph-mon[29810]: from='client.? 172.21.15.173:0/831357283' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:43:47.606 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:47 smithi173 ceph-mon[25952]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:47 smithi092 ceph-mon[25107]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:47.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:47 smithi092 ceph-mon[29810]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:49.646 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:49 smithi173 ceph-mon[25952]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:49.647 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-21T01:43:49.647 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:49.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:49 smithi092 ceph-mon[25107]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:49.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-21T01:43:49.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:49.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:49 smithi092 ceph-mon[29810]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:49.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-21T01:43:49.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:50.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:50 smithi173 ceph-mon[25952]: Deploying daemon osd.4 on smithi173 2024-06-21T01:43:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:50 smithi092 ceph-mon[25107]: Deploying daemon osd.4 on smithi173 2024-06-21T01:43:50.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:50 smithi092 ceph-mon[29810]: Deploying daemon osd.4 on smithi173 2024-06-21T01:43:51.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:51 smithi173 ceph-mon[25952]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:51 smithi092 ceph-mon[25107]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:51.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:51 smithi092 ceph-mon[29810]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:53.704 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:53 smithi173 ceph-mon[25952]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:53.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:53 smithi092 ceph-mon[25107]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:53.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:53 smithi092 ceph-mon[29810]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:55.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:54 smithi173 ceph-mon[25952]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:55.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:54 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:55.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:54 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:55.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:54 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:55.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:54 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:55.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:54 smithi092 ceph-mon[29810]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:55.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:55.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:55.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:55.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:54 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:55.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:54 smithi092 ceph-mon[25107]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:55.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:55.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:55.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:55.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:54 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:57.274 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 4 on host 'smithi173' 2024-06-21T01:43:57.554 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:57 smithi173 ceph-mon[25952]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:57.555 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:57 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:57.555 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:57 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:57.555 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:57 smithi173 ceph-mon[25952]: from='osd.4 [v2:172.21.15.173:6800/1078833784,v1:172.21.15.173:6801/1078833784]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:43:57.555 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:57 smithi173 ceph-mon[25952]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:43:57.555 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:57 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:57.555 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:57 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:57.555 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:57 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:57.555 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:57 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:57.844 DEBUG:teuthology.orchestra.run.smithi173:osd.4> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.4.service 2024-06-21T01:43:57.846 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/vg_nvme/lv_3... 2024-06-21T01:43:57.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-21T01:43:57.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[29810]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:57.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:57.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:57.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[29810]: from='osd.4 [v2:172.21.15.173:6800/1078833784,v1:172.21.15.173:6801/1078833784]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:43:57.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[29810]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:43:57.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:57.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:57.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:57.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:57.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[25107]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:57.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:57.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:57.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[25107]: from='osd.4 [v2:172.21.15.173:6800/1078833784,v1:172.21.15.173:6801/1078833784]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:43:57.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[25107]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:43:57.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:43:57.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:43:57.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:43:57.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:43:58.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:58 smithi173 ceph-mon[25952]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-21T01:43:58.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:58 smithi173 ceph-mon[25952]: osdmap e29: 5 total, 4 up, 5 in 2024-06-21T01:43:58.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:58 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:43:58.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:58 smithi173 ceph-mon[25952]: from='osd.4 [v2:172.21.15.173:6800/1078833784,v1:172.21.15.173:6801/1078833784]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:43:58.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:58 smithi173 ceph-mon[25952]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:43:58.729 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:43:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[29372]: 2024-06-21T01:43:58.421+0000 7f326940f700 -1 osd.4 0 waiting for initial osdmap 2024-06-21T01:43:58.729 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:43:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[29372]: 2024-06-21T01:43:58.426+0000 7f3265daa700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:43:58.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:58 smithi092 ceph-mon[25107]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-21T01:43:58.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:58 smithi092 ceph-mon[25107]: osdmap e29: 5 total, 4 up, 5 in 2024-06-21T01:43:58.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:58 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:43:58.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:58 smithi092 ceph-mon[25107]: from='osd.4 [v2:172.21.15.173:6800/1078833784,v1:172.21.15.173:6801/1078833784]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:43:58.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:58 smithi092 ceph-mon[25107]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:43:58.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:58 smithi092 ceph-mon[29810]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-21T01:43:58.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:58 smithi092 ceph-mon[29810]: osdmap e29: 5 total, 4 up, 5 in 2024-06-21T01:43:58.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:58 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:43:58.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:58 smithi092 ceph-mon[29810]: from='osd.4 [v2:172.21.15.173:6800/1078833784,v1:172.21.15.173:6801/1078833784]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:43:58.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:58 smithi092 ceph-mon[29810]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:43:59.676 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:59 smithi173 ceph-mon[25952]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:59.676 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:59 smithi173 ceph-mon[25952]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:43:59.676 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:59 smithi173 ceph-mon[25952]: osdmap e30: 5 total, 4 up, 5 in 2024-06-21T01:43:59.676 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:59 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:43:59.676 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:43:59 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:43:59.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:59 smithi092 ceph-mon[25107]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:59.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:59 smithi092 ceph-mon[25107]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:43:59.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:59 smithi092 ceph-mon[25107]: osdmap e30: 5 total, 4 up, 5 in 2024-06-21T01:43:59.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:59 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:43:59.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:43:59 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:43:59.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:59 smithi092 ceph-mon[29810]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-21T01:43:59.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:59 smithi092 ceph-mon[29810]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:43:59.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:59 smithi092 ceph-mon[29810]: osdmap e30: 5 total, 4 up, 5 in 2024-06-21T01:43:59.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:59 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:43:59.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:43:59 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:44:00.144 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-21T01:44:00.144 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-21T01:44:00.144 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-06-21T01:44:00.144 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-06-21T01:44:00.144 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012423 s, 844 MB/s 2024-06-21T01:44:00.144 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-06-21T01:44:00.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:00 smithi173 ceph-mon[25952]: purged_snaps scrub starts 2024-06-21T01:44:00.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:00 smithi173 ceph-mon[25952]: purged_snaps scrub ok 2024-06-21T01:44:00.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:00 smithi173 ceph-mon[25952]: osd.4 [v2:172.21.15.173:6800/1078833784,v1:172.21.15.173:6801/1078833784] boot 2024-06-21T01:44:00.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:00 smithi173 ceph-mon[25952]: osdmap e31: 5 total, 5 up, 5 in 2024-06-21T01:44:00.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:00 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:44:00.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:00 smithi173 ceph-mon[25952]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:00.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:00 smithi173 ceph-mon[25952]: osdmap e32: 5 total, 5 up, 5 in 2024-06-21T01:44:00.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-06-21T01:44:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[25107]: purged_snaps scrub starts 2024-06-21T01:44:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[25107]: purged_snaps scrub ok 2024-06-21T01:44:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[25107]: osd.4 [v2:172.21.15.173:6800/1078833784,v1:172.21.15.173:6801/1078833784] boot 2024-06-21T01:44:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[25107]: osdmap e31: 5 total, 5 up, 5 in 2024-06-21T01:44:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:44:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[25107]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:00.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[25107]: osdmap e32: 5 total, 5 up, 5 in 2024-06-21T01:44:00.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[29810]: purged_snaps scrub starts 2024-06-21T01:44:00.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[29810]: purged_snaps scrub ok 2024-06-21T01:44:00.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[29810]: osd.4 [v2:172.21.15.173:6800/1078833784,v1:172.21.15.173:6801/1078833784] boot 2024-06-21T01:44:00.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[29810]: osdmap e31: 5 total, 5 up, 5 in 2024-06-21T01:44:00.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:44:00.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[29810]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:00.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:00 smithi092 ceph-mon[29810]: osdmap e32: 5 total, 5 up, 5 in 2024-06-21T01:44:02.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:02 smithi173 ceph-mon[25952]: osdmap e33: 5 total, 5 up, 5 in 2024-06-21T01:44:02.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:02 smithi092 ceph-mon[25107]: osdmap e33: 5 total, 5 up, 5 in 2024-06-21T01:44:02.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:02 smithi092 ceph-mon[29810]: osdmap e33: 5 total, 5 up, 5 in 2024-06-21T01:44:03.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:03.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: osdmap e34: 5 total, 5 up, 5 in 2024-06-21T01:44:03.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:44:03.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:44:03.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:44:03.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:03.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: Detected new or changed devices on smithi173 2024-06-21T01:44:03.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:03.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:03.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: Adjusting osd_memory_target on smithi173 to 17088M 2024-06-21T01:44:03.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:03.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:03 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:03.752 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: osdmap e34: 5 total, 5 up, 5 in 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: Detected new or changed devices on smithi173 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: Adjusting osd_memory_target on smithi173 to 17088M 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:03.753 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:03.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: osdmap e34: 5 total, 5 up, 5 in 2024-06-21T01:44:03.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:44:03.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:44:03.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:44:03.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:03.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: Detected new or changed devices on smithi173 2024-06-21T01:44:03.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:03.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:03.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: Adjusting osd_memory_target on smithi173 to 17088M 2024-06-21T01:44:03.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:03.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:03 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:04.676 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:04 smithi173 ceph-mon[25952]: osdmap e35: 5 total, 5 up, 5 in 2024-06-21T01:44:04.676 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:04 smithi173 ceph-mon[25952]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-06-21T01:44:04.768 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:04 smithi092 ceph-mon[25107]: osdmap e35: 5 total, 5 up, 5 in 2024-06-21T01:44:04.768 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:04 smithi092 ceph-mon[25107]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-06-21T01:44:04.768 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:04 smithi092 ceph-mon[29810]: osdmap e35: 5 total, 5 up, 5 in 2024-06-21T01:44:04.768 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:04 smithi092 ceph-mon[29810]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-06-21T01:44:05.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:05 smithi173 ceph-mon[25952]: from='client.? 172.21.15.173:0/3646192657' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d573737f-e050-4cf0-a0b8-2176ea6ae476"}]: dispatch 2024-06-21T01:44:05.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:05 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d573737f-e050-4cf0-a0b8-2176ea6ae476"}]: dispatch 2024-06-21T01:44:05.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:05 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d573737f-e050-4cf0-a0b8-2176ea6ae476"}]': finished 2024-06-21T01:44:05.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:05 smithi173 ceph-mon[25952]: osdmap e36: 6 total, 5 up, 6 in 2024-06-21T01:44:05.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:05.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:05 smithi092 ceph-mon[29810]: from='client.? 172.21.15.173:0/3646192657' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d573737f-e050-4cf0-a0b8-2176ea6ae476"}]: dispatch 2024-06-21T01:44:05.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:05 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d573737f-e050-4cf0-a0b8-2176ea6ae476"}]: dispatch 2024-06-21T01:44:05.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:05 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d573737f-e050-4cf0-a0b8-2176ea6ae476"}]': finished 2024-06-21T01:44:05.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:05 smithi092 ceph-mon[29810]: osdmap e36: 6 total, 5 up, 6 in 2024-06-21T01:44:05.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:05.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:05 smithi092 ceph-mon[25107]: from='client.? 172.21.15.173:0/3646192657' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d573737f-e050-4cf0-a0b8-2176ea6ae476"}]: dispatch 2024-06-21T01:44:05.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:05 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d573737f-e050-4cf0-a0b8-2176ea6ae476"}]: dispatch 2024-06-21T01:44:05.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:05 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d573737f-e050-4cf0-a0b8-2176ea6ae476"}]': finished 2024-06-21T01:44:05.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:05 smithi092 ceph-mon[25107]: osdmap e36: 6 total, 5 up, 6 in 2024-06-21T01:44:05.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:06.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:06 smithi173 ceph-mon[25952]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-06-21T01:44:06.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:06 smithi173 ceph-mon[25952]: from='client.? 172.21.15.173:0/2666852099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:44:06.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:06 smithi092 ceph-mon[29810]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-06-21T01:44:06.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:06 smithi092 ceph-mon[29810]: from='client.? 172.21.15.173:0/2666852099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:44:06.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:06 smithi092 ceph-mon[25107]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-06-21T01:44:06.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:06 smithi092 ceph-mon[25107]: from='client.? 172.21.15.173:0/2666852099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:44:08.689 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:08 smithi173 ceph-mon[25952]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-21T01:44:08.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:08 smithi092 ceph-mon[29810]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-21T01:44:08.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:08 smithi092 ceph-mon[25107]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-21T01:44:09.720 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:09 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:44:09.721 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:09 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:44:09.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:09 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:44:09.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:09 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:44:09.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:09 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:44:09.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:09 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:44:10.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:10 smithi092 ceph-mon[29810]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-06-21T01:44:10.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:10 smithi092 ceph-mon[25107]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-06-21T01:44:10.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:10 smithi173 ceph-mon[25952]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-06-21T01:44:11.599 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:11 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-21T01:44:11.600 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:11 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:11.600 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:11 smithi173 ceph-mon[25952]: Deploying daemon osd.5 on smithi173 2024-06-21T01:44:11.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:11 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-21T01:44:11.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:11 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:11.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:11 smithi092 ceph-mon[29810]: Deploying daemon osd.5 on smithi173 2024-06-21T01:44:11.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:11 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-21T01:44:11.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:11 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:11.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:11 smithi092 ceph-mon[25107]: Deploying daemon osd.5 on smithi173 2024-06-21T01:44:13.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:13 smithi173 ceph-mon[25952]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:13.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:13 smithi092 ceph-mon[25107]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:13.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:13 smithi092 ceph-mon[29810]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:15.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:15 smithi173 ceph-mon[25952]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:15.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:15 smithi092 ceph-mon[25107]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:15.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:15 smithi092 ceph-mon[29810]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:17.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:17 smithi173 ceph-mon[25952]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:17.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:17 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:17.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:17 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:17.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:17 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:17.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:17 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:17.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:17 smithi092 ceph-mon[25107]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:17.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:17 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:17.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:17 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:17.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:17 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:17.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:17 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:17.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:17 smithi092 ceph-mon[29810]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:17.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:17 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:17.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:17 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:17.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:17 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:17.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:17 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:19.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:19 smithi173 ceph-mon[25952]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:19.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:19 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:19.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:19 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:19.273 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2024-06-21T01:44:19.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:19 smithi092 ceph-mon[25107]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:19.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:19 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:19.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:19 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:19.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:19 smithi092 ceph-mon[29810]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:19.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:19 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:19.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:19 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:20.017 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.5.service 2024-06-21T01:44:20.019 INFO:tasks.cephadm:Deploying osd.6 on smithi173 with /dev/vg_nvme/lv_2... 2024-06-21T01:44:20.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-21T01:44:20.309 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:20 smithi173 ceph-mon[25952]: from='osd.5 [v2:172.21.15.173:6808/3868923521,v1:172.21.15.173:6809/3868923521]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:44:20.309 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:20 smithi173 ceph-mon[25952]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:44:20.309 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:20 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:20.309 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:20 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:20.309 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:20 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:20.309 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:20 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:20.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[25107]: from='osd.5 [v2:172.21.15.173:6808/3868923521,v1:172.21.15.173:6809/3868923521]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:44:20.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[25107]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:44:20.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:20.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:20.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[29810]: from='osd.5 [v2:172.21.15.173:6808/3868923521,v1:172.21.15.173:6809/3868923521]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:44:20.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[29810]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:44:20.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:20.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:20.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:20.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:20 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:21.399 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:21 smithi173 ceph-mon[25952]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:21.399 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:21 smithi173 ceph-mon[25952]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-21T01:44:21.399 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:21 smithi173 ceph-mon[25952]: osdmap e37: 6 total, 5 up, 6 in 2024-06-21T01:44:21.399 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:21 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:21.399 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:21 smithi173 ceph-mon[25952]: from='osd.5 [v2:172.21.15.173:6808/3868923521,v1:172.21.15.173:6809/3868923521]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:21.399 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:21 smithi173 ceph-mon[25952]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:21.400 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:44:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[32238]: 2024-06-21T01:44:21.153+0000 7fc90aeda700 -1 osd.5 0 waiting for initial osdmap 2024-06-21T01:44:21.400 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:44:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[32238]: 2024-06-21T01:44:21.159+0000 7fc905871700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:44:21.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[25107]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:21.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[25107]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-21T01:44:21.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[25107]: osdmap e37: 6 total, 5 up, 6 in 2024-06-21T01:44:21.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:21.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[25107]: from='osd.5 [v2:172.21.15.173:6808/3868923521,v1:172.21.15.173:6809/3868923521]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:21.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[25107]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:21.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[29810]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:21.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[29810]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-21T01:44:21.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[29810]: osdmap e37: 6 total, 5 up, 6 in 2024-06-21T01:44:21.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:21.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[29810]: from='osd.5 [v2:172.21.15.173:6808/3868923521,v1:172.21.15.173:6809/3868923521]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:21.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:21 smithi092 ceph-mon[29810]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:22.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:22 smithi173 ceph-mon[25952]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:44:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:22 smithi173 ceph-mon[25952]: osdmap e38: 6 total, 5 up, 6 in 2024-06-21T01:44:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:22 smithi173 ceph-mon[25952]: osd.5 [v2:172.21.15.173:6808/3868923521,v1:172.21.15.173:6809/3868923521] boot 2024-06-21T01:44:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:22 smithi173 ceph-mon[25952]: osdmap e39: 6 total, 6 up, 6 in 2024-06-21T01:44:22.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:22 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:22.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[25107]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:44:22.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[25107]: osdmap e38: 6 total, 5 up, 6 in 2024-06-21T01:44:22.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:22.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:22.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[25107]: osd.5 [v2:172.21.15.173:6808/3868923521,v1:172.21.15.173:6809/3868923521] boot 2024-06-21T01:44:22.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[25107]: osdmap e39: 6 total, 6 up, 6 in 2024-06-21T01:44:22.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:22.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[29810]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:44:22.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[29810]: osdmap e38: 6 total, 5 up, 6 in 2024-06-21T01:44:22.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:22.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:22.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[29810]: osd.5 [v2:172.21.15.173:6808/3868923521,v1:172.21.15.173:6809/3868923521] boot 2024-06-21T01:44:22.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[29810]: osdmap e39: 6 total, 6 up, 6 in 2024-06-21T01:44:22.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:22 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:44:23.445 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:23 smithi173 ceph-mon[25952]: purged_snaps scrub starts 2024-06-21T01:44:23.445 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:23 smithi173 ceph-mon[25952]: purged_snaps scrub ok 2024-06-21T01:44:23.445 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:23 smithi173 ceph-mon[25952]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:23.445 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:23 smithi173 ceph-mon[25952]: osdmap e40: 6 total, 6 up, 6 in 2024-06-21T01:44:23.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:23 smithi092 ceph-mon[25107]: purged_snaps scrub starts 2024-06-21T01:44:23.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:23 smithi092 ceph-mon[25107]: purged_snaps scrub ok 2024-06-21T01:44:23.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:23 smithi092 ceph-mon[25107]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:23.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:23 smithi092 ceph-mon[25107]: osdmap e40: 6 total, 6 up, 6 in 2024-06-21T01:44:23.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:23 smithi092 ceph-mon[29810]: purged_snaps scrub starts 2024-06-21T01:44:23.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:23 smithi092 ceph-mon[29810]: purged_snaps scrub ok 2024-06-21T01:44:23.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:23 smithi092 ceph-mon[29810]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-21T01:44:23.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:23 smithi092 ceph-mon[29810]: osdmap e40: 6 total, 6 up, 6 in 2024-06-21T01:44:23.742 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-21T01:44:23.742 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-21T01:44:23.742 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-06-21T01:44:23.742 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-06-21T01:44:23.743 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148783 s, 705 MB/s 2024-06-21T01:44:23.743 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-06-21T01:44:24.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-06-21T01:44:24.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:24 smithi173 ceph-mon[25952]: osdmap e41: 6 total, 6 up, 6 in 2024-06-21T01:44:24.708 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:24 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:24.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:24 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:24.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:24 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:24.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:24 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:24.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:24 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:24.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[25107]: osdmap e41: 6 total, 6 up, 6 in 2024-06-21T01:44:24.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:24.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:24.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:24.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:24.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:24.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[29810]: osdmap e41: 6 total, 6 up, 6 in 2024-06-21T01:44:24.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:24.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:24.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:24.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:24.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:24 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:25.639 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:25 smithi173 ceph-mon[25952]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:25.639 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:25 smithi173 ceph-mon[25952]: Detected new or changed devices on smithi173 2024-06-21T01:44:25.640 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:25 smithi173 ceph-mon[25952]: Adjusting osd_memory_target on smithi173 to 8544M 2024-06-21T01:44:25.640 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:25 smithi173 ceph-mon[25952]: osdmap e42: 6 total, 6 up, 6 in 2024-06-21T01:44:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:25 smithi092 ceph-mon[25107]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:25 smithi092 ceph-mon[25107]: Detected new or changed devices on smithi173 2024-06-21T01:44:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:25 smithi092 ceph-mon[25107]: Adjusting osd_memory_target on smithi173 to 8544M 2024-06-21T01:44:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:25 smithi092 ceph-mon[25107]: osdmap e42: 6 total, 6 up, 6 in 2024-06-21T01:44:25.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:25 smithi092 ceph-mon[29810]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:25.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:25 smithi092 ceph-mon[29810]: Detected new or changed devices on smithi173 2024-06-21T01:44:25.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:25 smithi092 ceph-mon[29810]: Adjusting osd_memory_target on smithi173 to 8544M 2024-06-21T01:44:25.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:25 smithi092 ceph-mon[29810]: osdmap e42: 6 total, 6 up, 6 in 2024-06-21T01:44:26.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:26 smithi173 ceph-mon[25952]: osdmap e43: 6 total, 6 up, 6 in 2024-06-21T01:44:26.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:26 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:44:26.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:26 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:44:26.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:26 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:26 smithi092 ceph-mon[25107]: osdmap e43: 6 total, 6 up, 6 in 2024-06-21T01:44:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:26 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:44:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:26 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:44:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:26 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:26 smithi092 ceph-mon[29810]: osdmap e43: 6 total, 6 up, 6 in 2024-06-21T01:44:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:26 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:44:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:26 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:44:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:26 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:27.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:27 smithi173 ceph-mon[25952]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:27.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:27 smithi173 ceph-mon[25952]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:44:27.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:27 smithi092 ceph-mon[25107]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:27.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:27 smithi092 ceph-mon[25107]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:44:27.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:27 smithi092 ceph-mon[29810]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:27.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:27 smithi092 ceph-mon[29810]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:44:28.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:28 smithi173 ceph-mon[25952]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:28.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:28 smithi173 ceph-mon[25952]: from='client.? 172.21.15.173:0/2643887977' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f95f0e2-c07c-4788-99ec-802a25a0e854"}]: dispatch 2024-06-21T01:44:28.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:28 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f95f0e2-c07c-4788-99ec-802a25a0e854"}]: dispatch 2024-06-21T01:44:28.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:28 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f95f0e2-c07c-4788-99ec-802a25a0e854"}]': finished 2024-06-21T01:44:28.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:28 smithi173 ceph-mon[25952]: osdmap e44: 7 total, 6 up, 7 in 2024-06-21T01:44:28.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:28 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:28.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[25107]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:28.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[25107]: from='client.? 172.21.15.173:0/2643887977' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f95f0e2-c07c-4788-99ec-802a25a0e854"}]: dispatch 2024-06-21T01:44:28.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f95f0e2-c07c-4788-99ec-802a25a0e854"}]: dispatch 2024-06-21T01:44:28.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f95f0e2-c07c-4788-99ec-802a25a0e854"}]': finished 2024-06-21T01:44:28.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[25107]: osdmap e44: 7 total, 6 up, 7 in 2024-06-21T01:44:28.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:28.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[29810]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:28.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[29810]: from='client.? 172.21.15.173:0/2643887977' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f95f0e2-c07c-4788-99ec-802a25a0e854"}]: dispatch 2024-06-21T01:44:28.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f95f0e2-c07c-4788-99ec-802a25a0e854"}]: dispatch 2024-06-21T01:44:28.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f95f0e2-c07c-4788-99ec-802a25a0e854"}]': finished 2024-06-21T01:44:28.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[29810]: osdmap e44: 7 total, 6 up, 7 in 2024-06-21T01:44:28.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:28 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:29.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:29 smithi173 ceph-mon[25952]: from='client.? 172.21.15.173:0/301220439' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:44:29.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:29 smithi092 ceph-mon[25107]: from='client.? 172.21.15.173:0/301220439' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:44:29.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:29 smithi092 ceph-mon[29810]: from='client.? 172.21.15.173:0/301220439' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:44:30.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:30 smithi173 ceph-mon[25952]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:30.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:30 smithi092 ceph-mon[25107]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:30.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:30 smithi092 ceph-mon[29810]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:33.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:33 smithi173 ceph-mon[25952]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:33.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:33 smithi092 ceph-mon[25107]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:33.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:33 smithi092 ceph-mon[29810]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:34.648 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:34 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-21T01:44:34.649 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:34 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:34.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:34 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-21T01:44:34.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:34 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:34.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:34 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-21T01:44:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:34 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:35.696 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:35 smithi173 ceph-mon[25952]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:35.697 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:35 smithi173 ceph-mon[25952]: Deploying daemon osd.6 on smithi173 2024-06-21T01:44:35.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:35 smithi092 ceph-mon[25107]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:35.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:35 smithi092 ceph-mon[25107]: Deploying daemon osd.6 on smithi173 2024-06-21T01:44:35.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:35 smithi092 ceph-mon[29810]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:35.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:35 smithi092 ceph-mon[29810]: Deploying daemon osd.6 on smithi173 2024-06-21T01:44:37.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:37 smithi092 ceph-mon[25107]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:37.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:37 smithi092 ceph-mon[29810]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:37.616 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:37 smithi173 ceph-mon[25952]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:39.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:39 smithi173 ceph-mon[25952]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:39.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:39 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:39.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:39 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:39.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:39 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:39.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:39 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:39.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:39 smithi092 ceph-mon[29810]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:39.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:39 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:39.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:39 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:39.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:39 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:39.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:39 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:39.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:39 smithi092 ceph-mon[25107]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:39.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:39 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:39.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:39 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:39.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:39 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:39.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:39 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:41.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:41 smithi173 ceph-mon[25952]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:41.834 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 6 on host 'smithi173' 2024-06-21T01:44:41.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:41 smithi092 ceph-mon[25107]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:41.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:41 smithi092 ceph-mon[29810]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:42.417 DEBUG:teuthology.orchestra.run.smithi173:osd.6> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.6.service 2024-06-21T01:44:42.420 INFO:tasks.cephadm:Deploying osd.7 on smithi173 with /dev/vg_nvme/lv_1... 2024-06-21T01:44:42.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-21T01:44:42.948 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:42 smithi173 ceph-mon[25952]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:42.948 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:42 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:42.948 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:42 smithi173 ceph-mon[25952]: from='osd.6 [v2:172.21.15.173:6816/2408750708,v1:172.21.15.173:6817/2408750708]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:44:42.948 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:42 smithi173 ceph-mon[25952]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:44:43.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:42 smithi092 ceph-mon[29810]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:43.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:42 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:43.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:42 smithi092 ceph-mon[29810]: from='osd.6 [v2:172.21.15.173:6816/2408750708,v1:172.21.15.173:6817/2408750708]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:44:43.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:42 smithi092 ceph-mon[29810]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:44:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:42 smithi092 ceph-mon[25107]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:42 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:42 smithi092 ceph-mon[25107]: from='osd.6 [v2:172.21.15.173:6816/2408750708,v1:172.21.15.173:6817/2408750708]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:44:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:42 smithi092 ceph-mon[25107]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:44:44.071 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:43 smithi173 ceph-mon[25952]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-21T01:44:44.071 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:43 smithi173 ceph-mon[25952]: osdmap e45: 7 total, 6 up, 7 in 2024-06-21T01:44:44.072 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:43 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:44.072 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:43 smithi173 ceph-mon[25952]: from='osd.6 [v2:172.21.15.173:6816/2408750708,v1:172.21.15.173:6817/2408750708]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:44.072 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:43 smithi173 ceph-mon[25952]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:44.072 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:44:43 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[35118]: 2024-06-21T01:44:43.858+0000 7f7b64a56700 -1 osd.6 0 waiting for initial osdmap 2024-06-21T01:44:44.072 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:44:43 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[35118]: 2024-06-21T01:44:43.865+0000 7f7b5d3e9700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:44:44.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:43 smithi092 ceph-mon[25107]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-21T01:44:44.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:43 smithi092 ceph-mon[25107]: osdmap e45: 7 total, 6 up, 7 in 2024-06-21T01:44:44.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:43 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:44.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:43 smithi092 ceph-mon[25107]: from='osd.6 [v2:172.21.15.173:6816/2408750708,v1:172.21.15.173:6817/2408750708]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:44.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:43 smithi092 ceph-mon[25107]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:44.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:43 smithi092 ceph-mon[29810]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-21T01:44:44.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:43 smithi092 ceph-mon[29810]: osdmap e45: 7 total, 6 up, 7 in 2024-06-21T01:44:44.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:43 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:44.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:43 smithi092 ceph-mon[29810]: from='osd.6 [v2:172.21.15.173:6816/2408750708,v1:172.21.15.173:6817/2408750708]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:44.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:43 smithi092 ceph-mon[29810]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:44:44.638 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-21T01:44:44.638 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-21T01:44:44.638 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-06-21T01:44:44.638 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-06-21T01:44:44.638 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0139565 s, 751 MB/s 2024-06-21T01:44:44.638 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-06-21T01:44:44.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:44 smithi173 ceph-mon[25952]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:44.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:44 smithi173 ceph-mon[25952]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:44:44.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:44 smithi173 ceph-mon[25952]: osdmap e46: 7 total, 6 up, 7 in 2024-06-21T01:44:44.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:44 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:44.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:44 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:44.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:44 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:45.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-06-21T01:44:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[25107]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[25107]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:44:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[25107]: osdmap e46: 7 total, 6 up, 7 in 2024-06-21T01:44:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:45.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[29810]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:45.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[29810]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:44:45.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[29810]: osdmap e46: 7 total, 6 up, 7 in 2024-06-21T01:44:45.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:45.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:45.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:44 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:46.195 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:45 smithi173 ceph-mon[25952]: purged_snaps scrub starts 2024-06-21T01:44:46.195 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:45 smithi173 ceph-mon[25952]: purged_snaps scrub ok 2024-06-21T01:44:46.195 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:45 smithi173 ceph-mon[25952]: osd.6 [v2:172.21.15.173:6816/2408750708,v1:172.21.15.173:6817/2408750708] boot 2024-06-21T01:44:46.195 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:45 smithi173 ceph-mon[25952]: osdmap e47: 7 total, 7 up, 7 in 2024-06-21T01:44:46.195 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:45 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:46.195 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:45 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[25107]: purged_snaps scrub starts 2024-06-21T01:44:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[25107]: purged_snaps scrub ok 2024-06-21T01:44:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[25107]: osd.6 [v2:172.21.15.173:6816/2408750708,v1:172.21.15.173:6817/2408750708] boot 2024-06-21T01:44:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[25107]: osdmap e47: 7 total, 7 up, 7 in 2024-06-21T01:44:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:46.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[29810]: purged_snaps scrub starts 2024-06-21T01:44:46.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[29810]: purged_snaps scrub ok 2024-06-21T01:44:46.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[29810]: osd.6 [v2:172.21.15.173:6816/2408750708,v1:172.21.15.173:6817/2408750708] boot 2024-06-21T01:44:46.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[29810]: osdmap e47: 7 total, 7 up, 7 in 2024-06-21T01:44:46.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:46.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:45 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:44:47.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:46 smithi173 ceph-mon[25952]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:47.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:46 smithi173 ceph-mon[25952]: osdmap e48: 7 total, 7 up, 7 in 2024-06-21T01:44:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:46 smithi092 ceph-mon[25107]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:46 smithi092 ceph-mon[25107]: osdmap e48: 7 total, 7 up, 7 in 2024-06-21T01:44:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:46 smithi092 ceph-mon[29810]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-21T01:44:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:46 smithi092 ceph-mon[29810]: osdmap e48: 7 total, 7 up, 7 in 2024-06-21T01:44:48.025 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:47 smithi173 ceph-mon[25952]: osdmap e49: 7 total, 7 up, 7 in 2024-06-21T01:44:48.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:47 smithi092 ceph-mon[25107]: osdmap e49: 7 total, 7 up, 7 in 2024-06-21T01:44:48.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:47 smithi092 ceph-mon[29810]: osdmap e49: 7 total, 7 up, 7 in 2024-06-21T01:44:49.082 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:48 smithi173 ceph-mon[25952]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:44:49.082 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:48 smithi173 ceph-mon[25952]: osdmap e50: 7 total, 7 up, 7 in 2024-06-21T01:44:49.082 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:48 smithi173 ceph-mon[25952]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:44:49.082 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:48 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:44:49.082 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:48 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:44:49.082 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:48 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:49.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[25107]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:44:49.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[25107]: osdmap e50: 7 total, 7 up, 7 in 2024-06-21T01:44:49.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[25107]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:44:49.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:44:49.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:44:49.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:49.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[29810]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:44:49.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[29810]: osdmap e50: 7 total, 7 up, 7 in 2024-06-21T01:44:49.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[29810]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:44:49.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-21T01:44:49.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-21T01:44:49.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:48 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:49.954 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:49 smithi173 ceph-mon[25952]: osdmap e51: 7 total, 7 up, 7 in 2024-06-21T01:44:49.954 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:49.954 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:49.954 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:49.954 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:49.954 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:49.954 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:49.955 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:49.955 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:49.955 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:49.955 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:49 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:50.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[25107]: osdmap e51: 7 total, 7 up, 7 in 2024-06-21T01:44:50.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:50.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:50.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:50.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:50.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:50.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:50.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:50.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:50.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:50.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:50.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[29810]: osdmap e51: 7 total, 7 up, 7 in 2024-06-21T01:44:50.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:50.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:50.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:50.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:44:50.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:50.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:50.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:44:50.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:50.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:44:50.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:49 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:44:51.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:50 smithi173 ceph-mon[25952]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:44:51.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:50 smithi173 ceph-mon[25952]: Detected new or changed devices on smithi173 2024-06-21T01:44:51.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:50 smithi173 ceph-mon[25952]: Adjusting osd_memory_target on smithi173 to 5696M 2024-06-21T01:44:51.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:50 smithi092 ceph-mon[25107]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:44:51.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:50 smithi092 ceph-mon[25107]: Detected new or changed devices on smithi173 2024-06-21T01:44:51.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:50 smithi092 ceph-mon[25107]: Adjusting osd_memory_target on smithi173 to 5696M 2024-06-21T01:44:51.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:50 smithi092 ceph-mon[29810]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:44:51.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:50 smithi092 ceph-mon[29810]: Detected new or changed devices on smithi173 2024-06-21T01:44:51.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:50 smithi092 ceph-mon[29810]: Adjusting osd_memory_target on smithi173 to 5696M 2024-06-21T01:44:52.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:51 smithi173 ceph-mon[25952]: from='client.? 172.21.15.173:0/832874237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cac32d9-a8bc-45e0-94ef-caaef279d4d2"}]: dispatch 2024-06-21T01:44:52.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:51 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cac32d9-a8bc-45e0-94ef-caaef279d4d2"}]: dispatch 2024-06-21T01:44:52.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:51 smithi173 ceph-mon[25952]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cac32d9-a8bc-45e0-94ef-caaef279d4d2"}]': finished 2024-06-21T01:44:52.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:51 smithi173 ceph-mon[25952]: osdmap e52: 8 total, 7 up, 8 in 2024-06-21T01:44:52.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:51 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:44:52.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:51 smithi173 ceph-mon[25952]: from='client.? 172.21.15.173:0/2713162144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:44:52.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[25107]: from='client.? 172.21.15.173:0/832874237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cac32d9-a8bc-45e0-94ef-caaef279d4d2"}]: dispatch 2024-06-21T01:44:52.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cac32d9-a8bc-45e0-94ef-caaef279d4d2"}]: dispatch 2024-06-21T01:44:52.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[25107]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cac32d9-a8bc-45e0-94ef-caaef279d4d2"}]': finished 2024-06-21T01:44:52.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[25107]: osdmap e52: 8 total, 7 up, 8 in 2024-06-21T01:44:52.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:44:52.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[25107]: from='client.? 172.21.15.173:0/2713162144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:44:52.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[29810]: from='client.? 172.21.15.173:0/832874237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cac32d9-a8bc-45e0-94ef-caaef279d4d2"}]: dispatch 2024-06-21T01:44:52.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cac32d9-a8bc-45e0-94ef-caaef279d4d2"}]: dispatch 2024-06-21T01:44:52.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[29810]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cac32d9-a8bc-45e0-94ef-caaef279d4d2"}]': finished 2024-06-21T01:44:52.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[29810]: osdmap e52: 8 total, 7 up, 8 in 2024-06-21T01:44:52.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:44:52.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:51 smithi092 ceph-mon[29810]: from='client.? 172.21.15.173:0/2713162144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-21T01:44:53.719 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:53 smithi173 ceph-mon[25952]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-06-21T01:44:53.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:53 smithi092 ceph-mon[25107]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-06-21T01:44:53.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:53 smithi092 ceph-mon[29810]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-06-21T01:44:55.676 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:55 smithi173 ceph-mon[25952]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-21T01:44:55.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:55 smithi092 ceph-mon[25107]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-21T01:44:55.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:55 smithi092 ceph-mon[29810]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-21T01:44:57.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:57 smithi173 ceph-mon[25952]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-06-21T01:44:57.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:57 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-21T01:44:57.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:57 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:57.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:57 smithi092 ceph-mon[25107]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-06-21T01:44:57.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-21T01:44:57.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:57 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:57.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:57 smithi092 ceph-mon[29810]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-06-21T01:44:57.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-21T01:44:57.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:57 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:44:58.726 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:58 smithi173 ceph-mon[25952]: Deploying daemon osd.7 on smithi173 2024-06-21T01:44:58.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:58 smithi092 ceph-mon[25107]: Deploying daemon osd.7 on smithi173 2024-06-21T01:44:58.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:58 smithi092 ceph-mon[29810]: Deploying daemon osd.7 on smithi173 2024-06-21T01:44:59.519 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:44:59 smithi173 ceph-mon[25952]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-06-21T01:44:59.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:44:59 smithi092 ceph-mon[25107]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-06-21T01:44:59.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:44:59 smithi092 ceph-mon[29810]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-06-21T01:45:00.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:00 smithi173 ceph-mon[25952]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-21T01:45:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:00 smithi092 ceph-mon[25107]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-21T01:45:00.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:00 smithi092 ceph-mon[29810]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-21T01:45:02.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:02 smithi173 ceph-mon[25952]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:45:02.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:02 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:02.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:02 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:02.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:02 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:02.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:02 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:03.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:02 smithi092 ceph-mon[25107]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:45:03.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:02 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:03.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:02 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:03.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:02 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:03.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:02 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:03.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:02 smithi092 ceph-mon[29810]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:45:03.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:02 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:03.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:02 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:03.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:02 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:03.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:02 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:04.823 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 7 on host 'smithi173' 2024-06-21T01:45:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[25107]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:45:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[25107]: from='osd.7 [v2:172.21.15.173:6824/105685003,v1:172.21.15.173:6825/105685003]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[25107]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[29810]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[29810]: from='osd.7 [v2:172.21.15.173:6824/105685003,v1:172.21.15.173:6825/105685003]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:45:05.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[29810]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:45:05.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:05.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:05.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:05.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:05 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:05.437 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:05 smithi173 ceph-mon[25952]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:45:05.437 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:05.437 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:05.437 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:05 smithi173 ceph-mon[25952]: from='osd.7 [v2:172.21.15.173:6824/105685003,v1:172.21.15.173:6825/105685003]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:45:05.437 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:05 smithi173 ceph-mon[25952]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:45:05.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:05.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:05.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:05.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:05 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:05.460 DEBUG:teuthology.orchestra.run.smithi173:osd.7> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.7.service 2024-06-21T01:45:05.462 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-06-21T01:45:05.462 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd stat -f json 2024-06-21T01:45:06.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:06 smithi092 ceph-mon[29810]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-21T01:45:06.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:06 smithi092 ceph-mon[29810]: osdmap e53: 8 total, 7 up, 8 in 2024-06-21T01:45:06.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:06 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:06.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:06 smithi092 ceph-mon[29810]: from='osd.7 [v2:172.21.15.173:6824/105685003,v1:172.21.15.173:6825/105685003]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:45:06.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:06 smithi092 ceph-mon[29810]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:45:06.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:06 smithi092 ceph-mon[25107]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-21T01:45:06.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:06 smithi092 ceph-mon[25107]: osdmap e53: 8 total, 7 up, 8 in 2024-06-21T01:45:06.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:06 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:06.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:06 smithi092 ceph-mon[25107]: from='osd.7 [v2:172.21.15.173:6824/105685003,v1:172.21.15.173:6825/105685003]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:45:06.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:06 smithi092 ceph-mon[25107]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:45:06.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:06 smithi173 ceph-mon[25952]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-21T01:45:06.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:06 smithi173 ceph-mon[25952]: osdmap e53: 8 total, 7 up, 8 in 2024-06-21T01:45:06.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:06 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:06.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:06 smithi173 ceph-mon[25952]: from='osd.7 [v2:172.21.15.173:6824/105685003,v1:172.21.15.173:6825/105685003]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:45:06.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:06 smithi173 ceph-mon[25952]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:45:06.479 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:45:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[38091]: 2024-06-21T01:45:06.070+0000 7fcaf872b700 -1 osd.7 0 waiting for initial osdmap 2024-06-21T01:45:06.479 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:45:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[38091]: 2024-06-21T01:45:06.077+0000 7fcaf10be700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:45:07.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:07 smithi173 ceph-mon[25952]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:45:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:07 smithi173 ceph-mon[25952]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:45:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:07 smithi173 ceph-mon[25952]: osdmap e54: 8 total, 7 up, 8 in 2024-06-21T01:45:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:07 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:07 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:07 smithi173 ceph-mon[25952]: osd.7 [v2:172.21.15.173:6824/105685003,v1:172.21.15.173:6825/105685003] boot 2024-06-21T01:45:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:07 smithi173 ceph-mon[25952]: osdmap e55: 8 total, 8 up, 8 in 2024-06-21T01:45:07.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:07 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:07.317 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[29810]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:45:07.317 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[29810]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:45:07.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[29810]: osdmap e54: 8 total, 7 up, 8 in 2024-06-21T01:45:07.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:07.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:07.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[29810]: osd.7 [v2:172.21.15.173:6824/105685003,v1:172.21.15.173:6825/105685003] boot 2024-06-21T01:45:07.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[29810]: osdmap e55: 8 total, 8 up, 8 in 2024-06-21T01:45:07.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:07.318 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[25107]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-21T01:45:07.318 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[25107]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-21T01:45:07.318 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[25107]: osdmap e54: 8 total, 7 up, 8 in 2024-06-21T01:45:07.318 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:07.319 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:07.319 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[25107]: osd.7 [v2:172.21.15.173:6824/105685003,v1:172.21.15.173:6825/105685003] boot 2024-06-21T01:45:07.319 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[25107]: osdmap e55: 8 total, 8 up, 8 in 2024-06-21T01:45:07.319 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:07 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:07.511 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:45:08.163 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1718934306,"num_in_osds":8,"osd_in_since":1718934290,"num_remapped_pgs":1} 2024-06-21T01:45:08.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-21T01:45:08.390 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:08.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:08 smithi173 ceph-mon[25952]: purged_snaps scrub starts 2024-06-21T01:45:08.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:08 smithi173 ceph-mon[25952]: purged_snaps scrub ok 2024-06-21T01:45:08.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:08 smithi173 ceph-mon[25952]: osdmap e56: 8 total, 8 up, 8 in 2024-06-21T01:45:08.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:08 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/686000523' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-21T01:45:08.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:08.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:08.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:08 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:08.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[25107]: purged_snaps scrub starts 2024-06-21T01:45:08.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[25107]: purged_snaps scrub ok 2024-06-21T01:45:08.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[25107]: osdmap e56: 8 total, 8 up, 8 in 2024-06-21T01:45:08.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/686000523' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-21T01:45:08.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:08.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:08.743 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:08.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[29810]: purged_snaps scrub starts 2024-06-21T01:45:08.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[29810]: purged_snaps scrub ok 2024-06-21T01:45:08.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[29810]: osdmap e56: 8 total, 8 up, 8 in 2024-06-21T01:45:08.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/686000523' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-21T01:45:08.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:08.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:45:08.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:08.744 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:08 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:09.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:09 smithi092 ceph-mon[29810]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:09.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:09 smithi092 ceph-mon[29810]: Detected new or changed devices on smithi173 2024-06-21T01:45:09.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:09 smithi092 ceph-mon[29810]: Adjusting osd_memory_target on smithi173 to 4272M 2024-06-21T01:45:09.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:09 smithi092 ceph-mon[29810]: osdmap e57: 8 total, 8 up, 8 in 2024-06-21T01:45:09.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:09 smithi092 ceph-mon[25107]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:09.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:09 smithi092 ceph-mon[25107]: Detected new or changed devices on smithi173 2024-06-21T01:45:09.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:09 smithi092 ceph-mon[25107]: Adjusting osd_memory_target on smithi173 to 4272M 2024-06-21T01:45:09.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:09 smithi092 ceph-mon[25107]: osdmap e57: 8 total, 8 up, 8 in 2024-06-21T01:45:09.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:09 smithi173 ceph-mon[25952]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:09.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:09 smithi173 ceph-mon[25952]: Detected new or changed devices on smithi173 2024-06-21T01:45:09.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:09 smithi173 ceph-mon[25952]: Adjusting osd_memory_target on smithi173 to 4272M 2024-06-21T01:45:09.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:09 smithi173 ceph-mon[25952]: osdmap e57: 8 total, 8 up, 8 in 2024-06-21T01:45:10.258 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:45:10.258 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":58,"fsid":"22eef780-2f6f-11ef-bc9d-c7b262605968","created":"2024-06-21T01:40:16.829394+0000","modified":"2024-06-21T01:45:09.427684+0000","last_up_change":"2024-06-21T01:45:06.408663+0000","last_in_change":"2024-06-21T01:44:50.941522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-21T01:43:17.756109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1e18f46e-a84c-4d61-ab85-72135a2a758d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6803","nonce":81657251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6805","nonce":81657251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6809","nonce":81657251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6807","nonce":81657251}]},"public_addr":"172.21.15.92:6803/81657251","cluster_addr":"172.21.15.92:6805/81657251","heartbeat_back_addr":"172.21.15.92:6809/81657251","heartbeat_front_addr":"172.21.15.92:6807/81657251","state":["exists","up"]},{"osd":1,"uuid":"06f0cb8c-8380-46d1-bf8e-cddd9930f5c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6811","nonce":185612642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6813","nonce":185612642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6817","nonce":185612642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6815","nonce":185612642}]},"public_addr":"172.21.15.92:6811/185612642","cluster_addr":"172.21.15.92:6813/185612642","heartbeat_back_addr":"172.21.15.92:6817/185612642","heartbeat_front_addr":"172.21.15.92:6815/185612642","state":["exists","up"]},{"osd":2,"uuid":"51c9d336-d57f-449e-bbda-07d212c4224d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6819","nonce":2600120716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6821","nonce":2600120716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6825","nonce":2600120716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6823","nonce":2600120716}]},"public_addr":"172.21.15.92:6819/2600120716","cluster_addr":"172.21.15.92:6821/2600120716","heartbeat_back_addr":"172.21.15.92:6825/2600120716","heartbeat_front_addr":"172.21.15.92:6823/2600120716","state":["exists","up"]},{"osd":3,"uuid":"22670eae-7380-47c6-ab26-9e9382026568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6827","nonce":2261799524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6829","nonce":2261799524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6833","nonce":2261799524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6831","nonce":2261799524}]},"public_addr":"172.21.15.92:6827/2261799524","cluster_addr":"172.21.15.92:6829/2261799524","heartbeat_back_addr":"172.21.15.92:6833/2261799524","heartbeat_front_addr":"172.21.15.92:6831/2261799524","state":["exists","up"]},{"osd":4,"uuid":"5f6c0946-0039-494e-b7fb-92bbdfb459c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6801","nonce":1078833784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6803","nonce":1078833784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6807","nonce":1078833784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6805","nonce":1078833784}]},"public_addr":"172.21.15.173:6801/1078833784","cluster_addr":"172.21.15.173:6803/1078833784","heartbeat_back_addr":"172.21.15.173:6807/1078833784","heartbeat_front_addr":"172.21.15.173:6805/1078833784","state":["exists","up"]},{"osd":5,"uuid":"d573737f-e050-4cf0-a0b8-2176ea6ae476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6809","nonce":3868923521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6811","nonce":3868923521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6815","nonce":3868923521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6813","nonce":3868923521}]},"public_addr":"172.21.15.173:6809/3868923521","cluster_addr":"172.21.15.173:6811/3868923521","heartbeat_back_addr":"172.21.15.173:6815/3868923521","heartbeat_front_addr":"172.21.15.173:6813/3868923521","state":["exists","up"]},{"osd":6,"uuid":"0f95f0e2-c07c-4788-99ec-802a25a0e854","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6817","nonce":2408750708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6819","nonce":2408750708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6823","nonce":2408750708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6821","nonce":2408750708}]},"public_addr":"172.21.15.173:6817/2408750708","cluster_addr":"172.21.15.173:6819/2408750708","heartbeat_back_addr":"172.21.15.173:6823/2408750708","heartbeat_front_addr":"172.21.15.173:6821/2408750708","state":["exists","up"]},{"osd":7,"uuid":"7cac32d9-a8bc-45e0-94ef-caaef279d4d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6825","nonce":105685003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6827","nonce":105685003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6831","nonce":105685003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6829","nonce":105685003}]},"public_addr":"172.21.15.173:6825/105685003","cluster_addr":"172.21.15.173:6827/105685003","heartbeat_back_addr":"172.21.15.173:6831/105685003","heartbeat_front_addr":"172.21.15.173:6829/105685003","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:42:26.874295+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:42:49.985221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:43:13.846924+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:43:37.314913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:43:58.009420+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:44:20.247985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:44:43.210662+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:45:05.718159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.92:0/4223667356":"2024-06-22T01:41:09.394151+0000","172.21.15.92:6801/2316531354":"2024-06-22T01:41:09.394151+0000","172.21.15.92:6800/2316531354":"2024-06-22T01:41:09.394151+0000","172.21.15.92:0/4215008918":"2024-06-22T01:41:09.394151+0000","172.21.15.92:0/1379486465":"2024-06-22T01:41:09.394151+0000","172.21.15.92:0/3774566714":"2024-06-22T01:40:47.203827+0000","172.21.15.92:0/1521357991":"2024-06-22T01:40:47.203827+0000","172.21.15.92:0/2570230159":"2024-06-22T01:40:47.203827+0000","172.21.15.92:6800/648013728":"2024-06-22T01:40:47.203827+0000","172.21.15.92:6801/648013728":"2024-06-22T01:40:47.203827+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-21T01:45:10.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:10 smithi092 ceph-mon[25107]: osdmap e58: 8 total, 8 up, 8 in 2024-06-21T01:45:10.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:45:10.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:10 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:45:10.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:10 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2772848116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-21T01:45:10.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:10 smithi092 ceph-mon[29810]: osdmap e58: 8 total, 8 up, 8 in 2024-06-21T01:45:10.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:45:10.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:10 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:45:10.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:10 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2772848116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-21T01:45:10.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:10 smithi173 ceph-mon[25952]: osdmap e58: 8 total, 8 up, 8 in 2024-06-21T01:45:10.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:10 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:45:10.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:10 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:45:10.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:10 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2772848116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-21T01:45:10.982 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-21T01:43:17.756109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-06-21T01:45:10.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-21T01:45:11.198 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:11.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:11 smithi092 ceph-mon[25107]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:11.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:11 smithi092 ceph-mon[25107]: osdmap e59: 8 total, 8 up, 8 in 2024-06-21T01:45:11.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:11 smithi092 ceph-mon[29810]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:11.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:11 smithi092 ceph-mon[29810]: osdmap e59: 8 total, 8 up, 8 in 2024-06-21T01:45:11.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:11 smithi173 ceph-mon[25952]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:11.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:11 smithi173 ceph-mon[25952]: osdmap e59: 8 total, 8 up, 8 in 2024-06-21T01:45:12.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:12 smithi092 ceph-mon[25107]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:12.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:12 smithi092 ceph-mon[29810]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:12.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:12 smithi173 ceph-mon[25952]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:13.122 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2024-06-21T01:45:13.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:13 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2768697854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-21T01:45:13.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:13 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2768697854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-21T01:45:13.815 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:13 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2768697854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-21T01:45:13.838 INFO:tasks.cephadm:Adding prometheus.a on smithi173 2024-06-21T01:45:13.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch apply prometheus '1;smithi173=a' 2024-06-21T01:45:14.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:14 smithi173 ceph-mon[25952]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-21T01:45:14.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:14 smithi092 ceph-mon[25107]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-21T01:45:14.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:14 smithi092 ceph-mon[29810]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-21T01:45:15.690 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled prometheus update... 2024-06-21T01:45:16.232 DEBUG:teuthology.orchestra.run.smithi173:prometheus.a> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@prometheus.a.service 2024-06-21T01:45:16.237 INFO:tasks.cephadm:Adding node-exporter.a on smithi092 2024-06-21T01:45:16.237 INFO:tasks.cephadm:Adding node-exporter.b on smithi173 2024-06-21T01:45:16.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch apply node-exporter '2;smithi092=a;smithi173=b' 2024-06-21T01:45:16.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-mon[25952]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-06-21T01:45:16.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-mon[25952]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:16.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-mon[25952]: Saving service prometheus spec with placement smithi173=a;count:1 2024-06-21T01:45:16.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:16.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:16.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:16.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:16.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:16.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-21T01:45:17.020 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ignoring --setuser ceph since I am not root 2024-06-21T01:45:17.020 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ignoring --setgroup ceph since I am not root 2024-06-21T01:45:17.020 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mgr[25340]: -- 172.21.15.92:0/900414560 <== mon.0 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cc26cd4340 con 0x55cc26cad000 2024-06-21T01:45:17.020 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:16.897+0000 7fb7c1c53000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:45:17.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[25107]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-06-21T01:45:17.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[25107]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:17.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[25107]: Saving service prometheus spec with placement smithi173=a;count:1 2024-06-21T01:45:17.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:17.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:17.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:17.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:17.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:17.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-21T01:45:17.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[29810]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-06-21T01:45:17.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[29810]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:17.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[29810]: Saving service prometheus spec with placement smithi173=a;count:1 2024-06-21T01:45:17.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:17.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:17.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:17.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:17.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' 2024-06-21T01:45:17.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:16 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-21T01:45:17.026 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ignoring --setuser ceph since I am not root 2024-06-21T01:45:17.026 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ignoring --setgroup ceph since I am not root 2024-06-21T01:45:17.026 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-mgr[27133]: -- 172.21.15.173:0/611835114 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5653cd63e340 con 0x5653cd617000 2024-06-21T01:45:17.026 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:16.896+0000 7fc8efa24000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:45:17.026 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:17 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:17.028+0000 7fc8efa24000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:45:17.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:17 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:17.023+0000 7fb7c1c53000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:45:17.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:17 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:17.121+0000 7fb7c1c53000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:45:17.436 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:17 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:17.130+0000 7fc8efa24000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:45:17.690 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:17 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:17.460+0000 7fc8efa24000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:45:17.690 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:17 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:17.543+0000 7fc8efa24000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:45:17.726 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:17 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:17.424+0000 7fb7c1c53000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:45:17.726 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:17 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:17.505+0000 7fb7c1c53000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:45:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:17 smithi173 ceph-mon[25952]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-21T01:45:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:17 smithi173 ceph-mon[25952]: mgrmap e16: y(active, since 4m), standbys: x 2024-06-21T01:45:18.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:17 smithi092 ceph-mon[25107]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-21T01:45:18.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:17 smithi092 ceph-mon[25107]: mgrmap e16: y(active, since 4m), standbys: x 2024-06-21T01:45:18.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:17 smithi092 ceph-mon[29810]: from='mgr.14150 172.21.15.92:0/87372260' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-21T01:45:18.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:17 smithi092 ceph-mon[29810]: mgrmap e16: y(active, since 4m), standbys: x 2024-06-21T01:45:18.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:18.682+0000 7fc8efa24000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:45:18.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:18.845+0000 7fc8efa24000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:45:18.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:18.925+0000 7fc8efa24000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:45:19.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:18 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:18.614+0000 7fb7c1c53000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:45:19.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:18 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:18.776+0000 7fb7c1c53000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:45:19.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:18 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:18.860+0000 7fb7c1c53000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:45:19.599 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:19.172+0000 7fb7c1c53000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:45:19.599 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:19.262+0000 7fb7c1c53000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:45:19.599 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:19.356+0000 7fb7c1c53000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:45:19.653 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:19 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:19.236+0000 7fc8efa24000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:45:19.653 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:19 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:19.326+0000 7fc8efa24000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:45:19.653 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:19 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:19.416+0000 7fc8efa24000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:45:19.653 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:19 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:19.655+0000 7fc8efa24000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:45:19.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:19.602+0000 7fb7c1c53000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:45:20.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:20.037+0000 7fb7c1c53000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:45:20.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:20 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:20.092+0000 7fc8efa24000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:45:20.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:20.522+0000 7fb7c1c53000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:45:20.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:20.616+0000 7fb7c1c53000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:45:20.958 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:20 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:20.577+0000 7fc8efa24000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:45:20.958 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:20 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:20.669+0000 7fc8efa24000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:45:21.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:20 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:20.960+0000 7fc8efa24000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:45:21.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:21.049+0000 7fc8efa24000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:45:21.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:20.907+0000 7fb7c1c53000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:45:21.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:20.996+0000 7fb7c1c53000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:45:21.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:21.455+0000 7fc8efa24000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:45:21.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:21.585+0000 7fc8efa24000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:45:21.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:21.677+0000 7fc8efa24000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:45:21.737 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:21 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:21.398+0000 7fb7c1c53000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:45:21.737 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:21 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:21.531+0000 7fb7c1c53000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:45:21.737 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:21 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:21.624+0000 7fb7c1c53000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:45:22.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:21 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:21.739+0000 7fb7c1c53000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:45:22.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:21 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:21.830+0000 7fb7c1c53000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:45:22.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:21.790+0000 7fc8efa24000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:45:22.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:21.879+0000 7fc8efa24000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:45:22.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:22.420+0000 7fc8efa24000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:45:22.730 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:22.376+0000 7fb7c1c53000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:45:22.996 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:22.732+0000 7fb7c1c53000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:45:22.997 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:45:22] ENGINE Bus STARTING 2024-06-21T01:45:22.997 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-mon[25107]: Active manager daemon y restarted 2024-06-21T01:45:22.997 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-mon[25107]: Activating manager daemon y 2024-06-21T01:45:22.997 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-mon[25107]: osdmap e60: 8 total, 8 up, 8 in 2024-06-21T01:45:22.997 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-mon[25107]: Standby manager daemon x restarted 2024-06-21T01:45:22.997 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-mon[25107]: Standby manager daemon x started 2024-06-21T01:45:22.997 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-mon[29810]: Active manager daemon y restarted 2024-06-21T01:45:22.997 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-mon[29810]: Activating manager daemon y 2024-06-21T01:45:22.998 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-mon[29810]: osdmap e60: 8 total, 8 up, 8 in 2024-06-21T01:45:22.998 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-mon[29810]: Standby manager daemon x restarted 2024-06-21T01:45:22.998 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:22 smithi092 ceph-mon[29810]: Standby manager daemon x started 2024-06-21T01:45:23.058 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-mon[25952]: Active manager daemon y restarted 2024-06-21T01:45:23.058 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-mon[25952]: Activating manager daemon y 2024-06-21T01:45:23.058 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-mon[25952]: osdmap e60: 8 total, 8 up, 8 in 2024-06-21T01:45:23.058 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-mon[25952]: Standby manager daemon x restarted 2024-06-21T01:45:23.058 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-mon[25952]: Standby manager daemon x started 2024-06-21T01:45:23.059 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:22.781+0000 7fc8efa24000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:45:23.059 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: [21/Jun/2024:01:45:22] ENGINE Bus STARTING 2024-06-21T01:45:23.059 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: CherryPy Checker: 2024-06-21T01:45:23.059 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: The Application mounted at '' has an empty config. 2024-06-21T01:45:23.059 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: 2024-06-21T01:45:23.059 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: [21/Jun/2024:01:45:22] ENGINE Serving on http://:::9283 2024-06-21T01:45:23.059 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: [21/Jun/2024:01:45:22] ENGINE Bus STARTED 2024-06-21T01:45:23.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: CherryPy Checker: 2024-06-21T01:45:23.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: The Application mounted at '' has an empty config. 2024-06-21T01:45:23.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:45:23.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:45:23] ENGINE Serving on http://:::9283 2024-06-21T01:45:23.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:45:23] ENGINE Bus STARTED 2024-06-21T01:45:23.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:45:23] ENGINE Bus STARTING 2024-06-21T01:45:23.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:45:23] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:45:23.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:45:23] ENGINE Bus STARTED 2024-06-21T01:45:23.819 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled node-exporter update... 2024-06-21T01:45:23.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: mgrmap e17: y(active, starting, since 0.0576738s), standbys: x 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:45:23.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: Manager daemon y is now available 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:23.833 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:45:23.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:45:23.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:45:23.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:45:23.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: [21/Jun/2024:01:45:23] ENGINE Bus STARTING 2024-06-21T01:45:23.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: [21/Jun/2024:01:45:23] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:45:23.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: [21/Jun/2024:01:45:23] ENGINE Bus STARTED 2024-06-21T01:45:23.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:45:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:45:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: mgrmap e17: y(active, starting, since 0.0576738s), standbys: x 2024-06-21T01:45:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:45:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:45:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:45:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:45:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:45:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:45:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:45:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: Manager daemon y is now available 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:24.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:24.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:24.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:45:24.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:45:24.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:45:24.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:45:24.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: [21/Jun/2024:01:45:23] ENGINE Bus STARTING 2024-06-21T01:45:24.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: [21/Jun/2024:01:45:23] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:45:24.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: [21/Jun/2024:01:45:23] ENGINE Bus STARTED 2024-06-21T01:45:24.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: mgrmap e17: y(active, starting, since 0.0576738s), standbys: x 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:45:24.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/1382457608' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: Manager daemon y is now available 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:24.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:45:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:45:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:45:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:45:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: [21/Jun/2024:01:45:23] ENGINE Bus STARTING 2024-06-21T01:45:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: [21/Jun/2024:01:45:23] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:45:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: [21/Jun/2024:01:45:23] ENGINE Bus STARTED 2024-06-21T01:45:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:24.484 DEBUG:teuthology.orchestra.run.smithi092:node-exporter.a> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.a.service 2024-06-21T01:45:24.486 DEBUG:teuthology.orchestra.run.smithi173:node-exporter.b> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.b.service 2024-06-21T01:45:24.489 INFO:tasks.cephadm:Adding alertmanager.a on smithi092 2024-06-21T01:45:24.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch apply alertmanager '1;smithi092=a' 2024-06-21T01:45:25.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[25107]: mgrmap e18: y(active, since 1.07511s), standbys: x 2024-06-21T01:45:25.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[25107]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi092=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:25.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[25107]: Saving service node-exporter spec with placement smithi092=a;smithi173=b;count:2 2024-06-21T01:45:25.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[25107]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:25.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:25.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:25.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:25.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:25.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[29810]: mgrmap e18: y(active, since 1.07511s), standbys: x 2024-06-21T01:45:25.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[29810]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi092=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:25.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[29810]: Saving service node-exporter spec with placement smithi092=a;smithi173=b;count:2 2024-06-21T01:45:25.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[29810]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:25.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:25.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:25.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:25.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:25.041 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:24 smithi173 ceph-mon[25952]: mgrmap e18: y(active, since 1.07511s), standbys: x 2024-06-21T01:45:25.041 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:24 smithi173 ceph-mon[25952]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi092=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:25.041 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:24 smithi173 ceph-mon[25952]: Saving service node-exporter spec with placement smithi092=a;smithi173=b;count:2 2024-06-21T01:45:25.041 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:24 smithi173 ceph-mon[25952]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:25.042 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:25.042 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:25.042 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:25.042 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.186 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:26 smithi173 ceph-mon[25952]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:45:26.186 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:26 smithi173 ceph-mon[25952]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:26.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:26 smithi173 ceph-mon[25952]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:45:26.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:26 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:26 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:26 smithi173 ceph-mon[25952]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:45:26.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:26 smithi173 ceph-mon[25952]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:45:26.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:26 smithi173 ceph-mon[25952]: mgrmap e19: y(active, since 2s), standbys: x 2024-06-21T01:45:26.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:26 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:26 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.579 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled alertmanager update... 2024-06-21T01:45:26.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[25107]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:45:26.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[25107]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:26.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[25107]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:45:26.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[25107]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:45:26.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[25107]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:45:26.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[25107]: mgrmap e19: y(active, since 2s), standbys: x 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[29810]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[29810]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[29810]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[29810]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[29810]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[29810]: mgrmap e19: y(active, since 2s), standbys: x 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:26.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:26 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:27.291 DEBUG:teuthology.orchestra.run.smithi092:alertmanager.a> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@alertmanager.a.service 2024-06-21T01:45:27.294 INFO:tasks.cephadm:Adding grafana.a on smithi173 2024-06-21T01:45:27.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph orch apply grafana '1;smithi173=a' 2024-06-21T01:45:27.842 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:27 smithi173 ceph-mon[25952]: Deploying daemon prometheus.a on smithi173 2024-06-21T01:45:27.842 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:27 smithi173 ceph-mon[25952]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:27.842 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:27 smithi173 ceph-mon[25952]: Saving service alertmanager spec with placement smithi092=a;count:1 2024-06-21T01:45:27.842 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:27 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:27.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:27 smithi092 ceph-mon[25107]: Deploying daemon prometheus.a on smithi173 2024-06-21T01:45:27.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:27 smithi092 ceph-mon[25107]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:27.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:27 smithi092 ceph-mon[25107]: Saving service alertmanager spec with placement smithi092=a;count:1 2024-06-21T01:45:27.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:27 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:27.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:27 smithi092 ceph-mon[29810]: Deploying daemon prometheus.a on smithi173 2024-06-21T01:45:27.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:27 smithi092 ceph-mon[29810]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:27.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:27 smithi092 ceph-mon[29810]: Saving service alertmanager spec with placement smithi092=a;count:1 2024-06-21T01:45:27.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:27 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:28.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:28 smithi092 ceph-mon[25107]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:28.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:28 smithi092 ceph-mon[29810]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:28 smithi173 ceph-mon[25952]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:30.415 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled grafana update... 2024-06-21T01:45:30.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:30 smithi092 ceph-mon[25107]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:30.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:30 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:30.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:30 smithi092 ceph-mon[29810]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:30.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:30 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:30 smithi173 ceph-mon[25952]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:30 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:31.264 DEBUG:teuthology.orchestra.run.smithi173:grafana.a> sudo journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@grafana.a.service 2024-06-21T01:45:31.271 INFO:tasks.cephadm:Setting up client nodes... 2024-06-21T01:45:31.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-21T01:45:31.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:31 smithi092 ceph-mon[25107]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:31.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:31 smithi092 ceph-mon[25107]: Saving service grafana spec with placement smithi173=a;count:1 2024-06-21T01:45:31.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:31 smithi092 ceph-mon[29810]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:31.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:31 smithi092 ceph-mon[29810]: Saving service grafana spec with placement smithi173=a;count:1 2024-06-21T01:45:31.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:31 smithi173 ceph-mon[25952]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:45:31.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:31 smithi173 ceph-mon[25952]: Saving service grafana spec with placement smithi173=a;count:1 2024-06-21T01:45:32.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:32 smithi092 ceph-mon[25107]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:32.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:32 smithi092 ceph-mon[29810]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:32.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:32 smithi173 ceph-mon[25952]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:33.369 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2024-06-21T01:45:33.370 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQA923RmllEGFhAAp20rJHihacvKhGZNzjYpig== 2024-06-21T01:45:33.698 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:33 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1242479524' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-21T01:45:33.698 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:33 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1242479524' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-21T01:45:33.699 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:33 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1242479524' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-21T01:45:33.699 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:33 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1242479524' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-21T01:45:33.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:33 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1242479524' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-21T01:45:33.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:33 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1242479524' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-21T01:45:34.005 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-06-21T01:45:34.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-21T01:45:34.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-21T01:45:34.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-21T01:45:34.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:34 smithi173 ceph-mon[25952]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:35.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:34 smithi092 ceph-mon[25107]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:35.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:34 smithi092 ceph-mon[29810]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:36.787 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:36 smithi173 ceph-mon[25952]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:37.029 INFO:teuthology.orchestra.run.smithi173.stdout:[client.1] 2024-06-21T01:45:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQBB23RmFiXCARAArTBaJI/Mj1FekiDd+rWzpQ== 2024-06-21T01:45:37.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:36 smithi092 ceph-mon[25107]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:37.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:36 smithi092 ceph-mon[29810]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:37.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:37 smithi173 ceph-mon[25952]: from='client.? 172.21.15.173:0/1507138540' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-21T01:45:37.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:37 smithi173 ceph-mon[25952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-21T01:45:37.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:37 smithi173 ceph-mon[25952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-21T01:45:38.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:37 smithi092 ceph-mon[25107]: from='client.? 172.21.15.173:0/1507138540' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-21T01:45:38.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:37 smithi092 ceph-mon[25107]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-21T01:45:38.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:37 smithi092 ceph-mon[25107]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-21T01:45:38.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:37 smithi092 ceph-mon[29810]: from='client.? 172.21.15.173:0/1507138540' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-21T01:45:38.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:37 smithi092 ceph-mon[29810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-21T01:45:38.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:37 smithi092 ceph-mon[29810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-21T01:45:38.169 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-21T01:45:38.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-21T01:45:38.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-21T01:45:38.209 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-21T01:45:38.210 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-21T01:45:38.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph mgr dump --format=json 2024-06-21T01:45:38.439 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:38.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:38 smithi092 ceph-mon[25107]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:38 smithi092 ceph-mon[29810]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:38.893 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:38 smithi173 ceph-mon[25952]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:40.335 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:45:40.478 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 systemd[1]: Starting Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:45:40.839 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-mon[25952]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:40.840 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3269613428' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-21T01:45:40.840 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 podman[40977]: 2024-06-21 01:45:40.49578508 +0000 UTC m=+0.017981428 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-06-21T01:45:40.840 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 podman[40977]: 2024-06-21 01:45:40.62272326 +0000 UTC m=+0.144919597 container create 13533f3524e0c189a721bffbe01b9c1092f2f6756a484b9ada60b10a52dc73d1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:45:40.840 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 podman[40977]: 2024-06-21 01:45:40.8166606 +0000 UTC m=+0.338856951 container init 13533f3524e0c189a721bffbe01b9c1092f2f6756a484b9ada60b10a52dc73d1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:45:40.840 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 podman[40977]: 2024-06-21 01:45:40.819365932 +0000 UTC m=+0.341562275 container start 13533f3524e0c189a721bffbe01b9c1092f2f6756a484b9ada60b10a52dc73d1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:45:40.920 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:40 smithi092 ceph-mon[29810]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:40.920 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:40 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3269613428' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-21T01:45:40.920 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:40 smithi092 ceph-mon[25107]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:40.920 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:40 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3269613428' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-21T01:45:40.942 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"active_gid":14448,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":2887174273},{"type":"v1","addr":"172.21.15.92:6801","nonce":2887174273}]},"active_addr":"172.21.15.92:6801/2887174273","active_change":"2024-06-21T01:45:22.735753+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24323,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.92:8443/","prometheus":"http://172.21.15.92:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":3550345633}]},{"addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":3976715161}]},{"addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":779119699}]},{"addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":67584640}]}]}} 2024-06-21T01:45:40.947 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-21T01:45:40.947 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-21T01:45:40.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-21T01:45:41.168 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:41.229 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.842Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-06-21T01:45:41.229 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.843Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.843Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.843Z caller=main.go:518 level=info host_details="(Linux 5.14.0-460.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024 x86_64 smithi173 (none))" 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.843Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.843Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.844Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.845Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.845Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.848Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.848Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.991µs 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.848Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.848Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-06-21T01:45:41.230 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.848Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.128µs wal_replay_duration=134.488µs total_replay_duration=169.411µs 2024-06-21T01:45:41.231 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.848Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-06-21T01:45:41.231 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.848Z caller=main.go:947 level=info msg="TSDB started" 2024-06-21T01:45:41.231 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.848Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-06-21T01:45:41.231 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.863Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.704484ms db_storage=639ns remote_storage=1.304µs web_handler=279ns query_engine=650ns scrape=351.584µs scrape_sd=16.09µs notify=924ns notify_sd=1.924µs rules=14.082999ms 2024-06-21T01:45:41.231 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:45:40.863Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-06-21T01:45:41.231 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 bash[40977]: 13533f3524e0c189a721bffbe01b9c1092f2f6756a484b9ada60b10a52dc73d1 2024-06-21T01:45:41.231 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:45:40 smithi173 systemd[1]: Started Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:45:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:41 smithi173 ceph-mon[25952]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:41 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:41 smithi173 ceph-mon[25952]: Deploying daemon node-exporter.a on smithi092 2024-06-21T01:45:42.241 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:41 smithi092 ceph-mon[25107]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:42.241 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:41 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:42.241 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:41 smithi092 ceph-mon[25107]: Deploying daemon node-exporter.a on smithi092 2024-06-21T01:45:42.241 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:41 smithi092 ceph-mon[29810]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:42.241 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:41 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:42.241 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:41 smithi092 ceph-mon[29810]: Deploying daemon node-exporter.a on smithi092 2024-06-21T01:45:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:45:43.048 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":60,"fsid":"22eef780-2f6f-11ef-bc9d-c7b262605968","created":"2024-06-21T01:40:16.829394+0000","modified":"2024-06-21T01:45:22.735417+0000","last_up_change":"2024-06-21T01:45:06.408663+0000","last_in_change":"2024-06-21T01:44:50.941522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-21T01:43:17.756109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1e18f46e-a84c-4d61-ab85-72135a2a758d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6803","nonce":81657251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6805","nonce":81657251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6809","nonce":81657251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6807","nonce":81657251}]},"public_addr":"172.21.15.92:6803/81657251","cluster_addr":"172.21.15.92:6805/81657251","heartbeat_back_addr":"172.21.15.92:6809/81657251","heartbeat_front_addr":"172.21.15.92:6807/81657251","state":["exists","up"]},{"osd":1,"uuid":"06f0cb8c-8380-46d1-bf8e-cddd9930f5c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6811","nonce":185612642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6813","nonce":185612642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6817","nonce":185612642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6815","nonce":185612642}]},"public_addr":"172.21.15.92:6811/185612642","cluster_addr":"172.21.15.92:6813/185612642","heartbeat_back_addr":"172.21.15.92:6817/185612642","heartbeat_front_addr":"172.21.15.92:6815/185612642","state":["exists","up"]},{"osd":2,"uuid":"51c9d336-d57f-449e-bbda-07d212c4224d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6819","nonce":2600120716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6821","nonce":2600120716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6825","nonce":2600120716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6823","nonce":2600120716}]},"public_addr":"172.21.15.92:6819/2600120716","cluster_addr":"172.21.15.92:6821/2600120716","heartbeat_back_addr":"172.21.15.92:6825/2600120716","heartbeat_front_addr":"172.21.15.92:6823/2600120716","state":["exists","up"]},{"osd":3,"uuid":"22670eae-7380-47c6-ab26-9e9382026568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6827","nonce":2261799524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6829","nonce":2261799524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6833","nonce":2261799524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6831","nonce":2261799524}]},"public_addr":"172.21.15.92:6827/2261799524","cluster_addr":"172.21.15.92:6829/2261799524","heartbeat_back_addr":"172.21.15.92:6833/2261799524","heartbeat_front_addr":"172.21.15.92:6831/2261799524","state":["exists","up"]},{"osd":4,"uuid":"5f6c0946-0039-494e-b7fb-92bbdfb459c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6801","nonce":1078833784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6803","nonce":1078833784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6807","nonce":1078833784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6805","nonce":1078833784}]},"public_addr":"172.21.15.173:6801/1078833784","cluster_addr":"172.21.15.173:6803/1078833784","heartbeat_back_addr":"172.21.15.173:6807/1078833784","heartbeat_front_addr":"172.21.15.173:6805/1078833784","state":["exists","up"]},{"osd":5,"uuid":"d573737f-e050-4cf0-a0b8-2176ea6ae476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6809","nonce":3868923521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6811","nonce":3868923521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6815","nonce":3868923521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6813","nonce":3868923521}]},"public_addr":"172.21.15.173:6809/3868923521","cluster_addr":"172.21.15.173:6811/3868923521","heartbeat_back_addr":"172.21.15.173:6815/3868923521","heartbeat_front_addr":"172.21.15.173:6813/3868923521","state":["exists","up"]},{"osd":6,"uuid":"0f95f0e2-c07c-4788-99ec-802a25a0e854","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6817","nonce":2408750708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6819","nonce":2408750708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6823","nonce":2408750708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6821","nonce":2408750708}]},"public_addr":"172.21.15.173:6817/2408750708","cluster_addr":"172.21.15.173:6819/2408750708","heartbeat_back_addr":"172.21.15.173:6823/2408750708","heartbeat_front_addr":"172.21.15.173:6821/2408750708","state":["exists","up"]},{"osd":7,"uuid":"7cac32d9-a8bc-45e0-94ef-caaef279d4d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6825","nonce":105685003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6827","nonce":105685003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6831","nonce":105685003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6829","nonce":105685003}]},"public_addr":"172.21.15.173:6825/105685003","cluster_addr":"172.21.15.173:6827/105685003","heartbeat_back_addr":"172.21.15.173:6831/105685003","heartbeat_front_addr":"172.21.15.173:6829/105685003","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:42:26.874295+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:42:49.985221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:43:13.846924+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:43:37.314913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:43:58.009420+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:44:20.247985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:44:43.210662+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:45:05.718159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.92:6800/4190712464":"2024-06-22T01:45:22.735384+0000","172.21.15.92:6801/4190712464":"2024-06-22T01:45:22.735384+0000","172.21.15.92:0/4042387550":"2024-06-22T01:45:22.735384+0000","172.21.15.92:0/333757986":"2024-06-22T01:45:22.735384+0000","172.21.15.92:0/244652261":"2024-06-22T01:45:22.735384+0000","172.21.15.92:0/1798744344":"2024-06-22T01:45:22.735384+0000","172.21.15.92:0/4223667356":"2024-06-22T01:41:09.394151+0000","172.21.15.92:6801/2316531354":"2024-06-22T01:41:09.394151+0000","172.21.15.92:6800/2316531354":"2024-06-22T01:41:09.394151+0000","172.21.15.92:0/4215008918":"2024-06-22T01:41:09.394151+0000","172.21.15.92:0/1379486465":"2024-06-22T01:41:09.394151+0000","172.21.15.92:0/3774566714":"2024-06-22T01:40:47.203827+0000","172.21.15.92:0/1521357991":"2024-06-22T01:40:47.203827+0000","172.21.15.92:0/2570230159":"2024-06-22T01:40:47.203827+0000","172.21.15.92:6800/648013728":"2024-06-22T01:40:47.203827+0000","172.21.15.92:6801/648013728":"2024-06-22T01:40:47.203827+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-21T01:45:43.048 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:42 smithi092 bash[44600]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-06-21T01:45:43.749 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-21T01:45:43.749 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-21T01:45:43.970 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:44.093 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:43 smithi092 ceph-mon[25107]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:44.093 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:43 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:44.093 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:43 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2527441392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-21T01:45:44.093 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:43 smithi092 ceph-mon[29810]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:44.093 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:43 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:44.093 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:43 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2527441392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-21T01:45:44.093 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:43 smithi092 bash[44600]: Getting image source signatures 2024-06-21T01:45:44.094 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:43 smithi092 bash[44600]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-06-21T01:45:44.094 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:43 smithi092 bash[44600]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-06-21T01:45:44.094 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:43 smithi092 bash[44600]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-06-21T01:45:44.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:43 smithi173 ceph-mon[25952]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:44.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:43 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:44.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:43 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2527441392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-21T01:45:45.480 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:45 smithi092 bash[44600]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-06-21T01:45:45.481 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:45 smithi092 bash[44600]: Writing manifest to image destination 2024-06-21T01:45:46.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:45 smithi092 ceph-mon[25107]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:46.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:45 smithi092 ceph-mon[29810]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:46.102 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:45 smithi092 podman[44600]: 2024-06-21 01:45:45.77422572 +0000 UTC m=+2.935618840 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-06-21T01:45:46.102 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:45 smithi092 podman[44600]: 2024-06-21 01:45:45.998192309 +0000 UTC m=+3.159585423 container create 71773b5f07f016cef897ab4d8cbf8dcf8c6e99a553a0d712112c79a398849116 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T01:45:46.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:45 smithi173 ceph-mon[25952]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:46.456 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 podman[44600]: 2024-06-21 01:45:46.453253889 +0000 UTC m=+3.614647017 container init 71773b5f07f016cef897ab4d8cbf8dcf8c6e99a553a0d712112c79a398849116 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T01:45:46.456 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 podman[44600]: 2024-06-21 01:45:46.456293956 +0000 UTC m=+3.617687071 container start 71773b5f07f016cef897ab4d8cbf8dcf8c6e99a553a0d712112c79a398849116 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T01:45:46.726 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.460Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-06-21T01:45:46.726 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.460Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-06-21T01:45:46.726 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.460Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-06-21T01:45:46.726 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-21T01:45:46.726 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-06-21T01:45:46.726 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=arp 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=bcache 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=bonding 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=btrfs 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=conntrack 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=cpu 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=diskstats 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=dmi 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=edac 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=entropy 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=filefd 2024-06-21T01:45:46.727 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=filesystem 2024-06-21T01:45:46.728 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=hwmon 2024-06-21T01:45:46.728 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=infiniband 2024-06-21T01:45:46.728 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=ipvs 2024-06-21T01:45:46.728 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=loadavg 2024-06-21T01:45:46.728 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=mdadm 2024-06-21T01:45:46.728 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=meminfo 2024-06-21T01:45:46.728 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=netclass 2024-06-21T01:45:46.728 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=netdev 2024-06-21T01:45:46.728 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=netstat 2024-06-21T01:45:46.728 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=nfs 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=nfsd 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=nvme 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=os 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=pressure 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=rapl 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=schedstat 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=sockstat 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=softnet 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=stat 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=tapestats 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=textfile 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=time 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-06-21T01:45:46.729 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=uname 2024-06-21T01:45:46.730 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=vmstat 2024-06-21T01:45:46.730 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=xfs 2024-06-21T01:45:46.730 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:115 level=info collector=zfs 2024-06-21T01:45:46.730 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-06-21T01:45:46.730 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[44766]: ts=2024-06-21T01:45:46.461Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-06-21T01:45:46.730 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 bash[44600]: 71773b5f07f016cef897ab4d8cbf8dcf8c6e99a553a0d712112c79a398849116 2024-06-21T01:45:46.730 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:45:46 smithi092 systemd[1]: Started Ceph node-exporter.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:45:47.006 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:45:47.006 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":60,"fsid":"22eef780-2f6f-11ef-bc9d-c7b262605968","created":"2024-06-21T01:40:16.829394+0000","modified":"2024-06-21T01:45:22.735417+0000","last_up_change":"2024-06-21T01:45:06.408663+0000","last_in_change":"2024-06-21T01:44:50.941522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-21T01:43:17.756109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"1e18f46e-a84c-4d61-ab85-72135a2a758d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6803","nonce":81657251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6805","nonce":81657251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6809","nonce":81657251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":81657251},{"type":"v1","addr":"172.21.15.92:6807","nonce":81657251}]},"public_addr":"172.21.15.92:6803/81657251","cluster_addr":"172.21.15.92:6805/81657251","heartbeat_back_addr":"172.21.15.92:6809/81657251","heartbeat_front_addr":"172.21.15.92:6807/81657251","state":["exists","up"]},{"osd":1,"uuid":"06f0cb8c-8380-46d1-bf8e-cddd9930f5c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6811","nonce":185612642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6813","nonce":185612642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6817","nonce":185612642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":185612642},{"type":"v1","addr":"172.21.15.92:6815","nonce":185612642}]},"public_addr":"172.21.15.92:6811/185612642","cluster_addr":"172.21.15.92:6813/185612642","heartbeat_back_addr":"172.21.15.92:6817/185612642","heartbeat_front_addr":"172.21.15.92:6815/185612642","state":["exists","up"]},{"osd":2,"uuid":"51c9d336-d57f-449e-bbda-07d212c4224d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6819","nonce":2600120716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6821","nonce":2600120716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6825","nonce":2600120716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":2600120716},{"type":"v1","addr":"172.21.15.92:6823","nonce":2600120716}]},"public_addr":"172.21.15.92:6819/2600120716","cluster_addr":"172.21.15.92:6821/2600120716","heartbeat_back_addr":"172.21.15.92:6825/2600120716","heartbeat_front_addr":"172.21.15.92:6823/2600120716","state":["exists","up"]},{"osd":3,"uuid":"22670eae-7380-47c6-ab26-9e9382026568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6827","nonce":2261799524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6829","nonce":2261799524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6833","nonce":2261799524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2261799524},{"type":"v1","addr":"172.21.15.92:6831","nonce":2261799524}]},"public_addr":"172.21.15.92:6827/2261799524","cluster_addr":"172.21.15.92:6829/2261799524","heartbeat_back_addr":"172.21.15.92:6833/2261799524","heartbeat_front_addr":"172.21.15.92:6831/2261799524","state":["exists","up"]},{"osd":4,"uuid":"5f6c0946-0039-494e-b7fb-92bbdfb459c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6801","nonce":1078833784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6803","nonce":1078833784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6807","nonce":1078833784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1078833784},{"type":"v1","addr":"172.21.15.173:6805","nonce":1078833784}]},"public_addr":"172.21.15.173:6801/1078833784","cluster_addr":"172.21.15.173:6803/1078833784","heartbeat_back_addr":"172.21.15.173:6807/1078833784","heartbeat_front_addr":"172.21.15.173:6805/1078833784","state":["exists","up"]},{"osd":5,"uuid":"d573737f-e050-4cf0-a0b8-2176ea6ae476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6809","nonce":3868923521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6811","nonce":3868923521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6815","nonce":3868923521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3868923521},{"type":"v1","addr":"172.21.15.173:6813","nonce":3868923521}]},"public_addr":"172.21.15.173:6809/3868923521","cluster_addr":"172.21.15.173:6811/3868923521","heartbeat_back_addr":"172.21.15.173:6815/3868923521","heartbeat_front_addr":"172.21.15.173:6813/3868923521","state":["exists","up"]},{"osd":6,"uuid":"0f95f0e2-c07c-4788-99ec-802a25a0e854","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6817","nonce":2408750708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6819","nonce":2408750708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6823","nonce":2408750708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2408750708},{"type":"v1","addr":"172.21.15.173:6821","nonce":2408750708}]},"public_addr":"172.21.15.173:6817/2408750708","cluster_addr":"172.21.15.173:6819/2408750708","heartbeat_back_addr":"172.21.15.173:6823/2408750708","heartbeat_front_addr":"172.21.15.173:6821/2408750708","state":["exists","up"]},{"osd":7,"uuid":"7cac32d9-a8bc-45e0-94ef-caaef279d4d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6825","nonce":105685003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6827","nonce":105685003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6831","nonce":105685003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":105685003},{"type":"v1","addr":"172.21.15.173:6829","nonce":105685003}]},"public_addr":"172.21.15.173:6825/105685003","cluster_addr":"172.21.15.173:6827/105685003","heartbeat_back_addr":"172.21.15.173:6831/105685003","heartbeat_front_addr":"172.21.15.173:6829/105685003","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:42:26.874295+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:42:49.985221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:43:13.846924+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:43:37.314913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:43:58.009420+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:44:20.247985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:44:43.210662+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-21T01:45:05.718159+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.92:6800/4190712464":"2024-06-22T01:45:22.735384+0000","172.21.15.92:6801/4190712464":"2024-06-22T01:45:22.735384+0000","172.21.15.92:0/4042387550":"2024-06-22T01:45:22.735384+0000","172.21.15.92:0/333757986":"2024-06-22T01:45:22.735384+0000","172.21.15.92:0/244652261":"2024-06-22T01:45:22.735384+0000","172.21.15.92:0/1798744344":"2024-06-22T01:45:22.735384+0000","172.21.15.92:0/4223667356":"2024-06-22T01:41:09.394151+0000","172.21.15.92:6801/2316531354":"2024-06-22T01:41:09.394151+0000","172.21.15.92:6800/2316531354":"2024-06-22T01:41:09.394151+0000","172.21.15.92:0/4215008918":"2024-06-22T01:41:09.394151+0000","172.21.15.92:0/1379486465":"2024-06-22T01:41:09.394151+0000","172.21.15.92:0/3774566714":"2024-06-22T01:40:47.203827+0000","172.21.15.92:0/1521357991":"2024-06-22T01:40:47.203827+0000","172.21.15.92:0/2570230159":"2024-06-22T01:40:47.203827+0000","172.21.15.92:6800/648013728":"2024-06-22T01:40:47.203827+0000","172.21.15.92:6801/648013728":"2024-06-22T01:40:47.203827+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-21T01:45:47.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-21T01:45:47.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-21T01:45:47.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-21T01:45:47.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-21T01:45:47.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-21T01:45:47.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-21T01:45:47.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-21T01:45:47.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-21T01:45:47.623 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:47 smithi173 bash[41212]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-06-21T01:45:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:47 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:47 smithi092 ceph-mon[25107]: Deploying daemon node-exporter.b on smithi173 2024-06-21T01:45:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:47 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2479797392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-21T01:45:47.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:47 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:47.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:47 smithi092 ceph-mon[29810]: Deploying daemon node-exporter.b on smithi173 2024-06-21T01:45:47.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:47 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2479797392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-21T01:45:47.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:47 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:47.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:47 smithi173 ceph-mon[25952]: Deploying daemon node-exporter.b on smithi173 2024-06-21T01:45:47.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:47 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2479797392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-21T01:45:47.991 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:47.991 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:47.993 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:47.994 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:47.997 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:47.998 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:47.998 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:48.004 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:45:48.478 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:48 smithi173 bash[41212]: Getting image source signatures 2024-06-21T01:45:48.478 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:48 smithi173 bash[41212]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-06-21T01:45:48.479 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:48 smithi173 bash[41212]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-06-21T01:45:48.479 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:48 smithi173 bash[41212]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-06-21T01:45:48.764 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:48 smithi092 ceph-mon[25107]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:48.764 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:48 smithi092 ceph-mon[29810]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:48.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:48 smithi173 ceph-mon[25952]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:49.978 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:49 smithi173 bash[41212]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-06-21T01:45:49.979 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:49 smithi173 bash[41212]: Writing manifest to image destination 2024-06-21T01:45:50.377 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 podman[41212]: 2024-06-21 01:45:50.106224907 +0000 UTC m=+2.728405546 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-06-21T01:45:50.378 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 podman[41212]: 2024-06-21 01:45:50.221296792 +0000 UTC m=+2.843477430 container create 8455f1de1751d1702c53fa53d0a685676c537d0c0b708e3619e2346d2579a11e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T01:45:50.378 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 podman[41212]: 2024-06-21 01:45:50.381601327 +0000 UTC m=+3.003781968 container init 8455f1de1751d1702c53fa53d0a685676c537d0c0b708e3619e2346d2579a11e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T01:45:50.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-mon[25952]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:50.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:50.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:50.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:50.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:50.730 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 podman[41212]: 2024-06-21 01:45:50.384636519 +0000 UTC m=+3.006817158 container start 8455f1de1751d1702c53fa53d0a685676c537d0c0b708e3619e2346d2579a11e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T01:45:50.730 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.388Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-06-21T01:45:50.730 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.388Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-06-21T01:45:50.730 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.388Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-06-21T01:45:50.730 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.388Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-21T01:45:50.730 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-06-21T01:45:50.730 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=arp 2024-06-21T01:45:50.730 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=bcache 2024-06-21T01:45:50.730 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=bonding 2024-06-21T01:45:50.730 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=btrfs 2024-06-21T01:45:50.730 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=conntrack 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=cpu 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=diskstats 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=dmi 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=edac 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=entropy 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=filefd 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=filesystem 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=hwmon 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=infiniband 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=ipvs 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=loadavg 2024-06-21T01:45:50.731 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=mdadm 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=meminfo 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=netclass 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=netdev 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=netstat 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=nfs 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=nfsd 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=nvme 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=os 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=pressure 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=rapl 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=schedstat 2024-06-21T01:45:50.732 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=sockstat 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=softnet 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=stat 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=tapestats 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=textfile 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=time 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=uname 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=vmstat 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=xfs 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:115 level=info collector=zfs 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.389Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[41269]: ts=2024-06-21T01:45:50.390Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 bash[41212]: 8455f1de1751d1702c53fa53d0a685676c537d0c0b708e3619e2346d2579a11e 2024-06-21T01:45:50.733 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:45:50 smithi173 systemd[1]: Started Ceph node-exporter.b for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:45:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:50 smithi092 ceph-mon[25107]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:50 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:50 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:50 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:50 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:50.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:50 smithi092 ceph-mon[29810]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:50 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:50 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:45:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:50 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:45:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:50 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:45:52.627 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:45:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:45:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:45:52.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:52 smithi092 ceph-mon[25107]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:52.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:52 smithi092 ceph-mon[29810]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:52.953 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:52 smithi173 ceph-mon[25952]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:53.995 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:53 smithi092 ceph-mon[25107]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:53.995 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:53 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:53.995 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:53 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:53.996 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:45:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:45:53] "GET /metrics HTTP/1.1" 200 191503 "" "Prometheus/2.33.4" 2024-06-21T01:45:53.996 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:53 smithi092 ceph-mon[29810]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:53.996 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:53 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:53.996 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:53 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:54.019 INFO:teuthology.orchestra.run.smithi092.stdout:77309411361 2024-06-21T01:45:54.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-21T01:45:54.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:53 smithi173 ceph-mon[25952]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:54.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:53 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:54.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:53 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:45:56.076 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:55 smithi092 ceph-mon[25107]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:56.076 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:55 smithi092 ceph-mon[29810]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:56.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:55 smithi173 ceph-mon[25952]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:56.981 INFO:teuthology.orchestra.run.smithi092.stdout:236223201291 2024-06-21T01:45:56.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-21T01:45:57.862 INFO:teuthology.orchestra.run.smithi092.stdout:34359738412 2024-06-21T01:45:57.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-21T01:45:58.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:45:58 smithi092 ceph-mon[25107]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:58.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:45:58 smithi092 ceph-mon[29810]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:45:58.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:45:58 smithi173 ceph-mon[25952]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:00.524 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:00 smithi092 ceph-mon[25107]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:00.524 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:00 smithi092 ceph-mon[29810]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:00.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:00 smithi173 ceph-mon[25952]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:01.460 INFO:teuthology.orchestra.run.smithi092.stdout:111669149727 2024-06-21T01:46:01.461 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-21T01:46:01.671 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:46:02.106 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:46:02.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:02 smithi092 ceph-mon[25107]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:02.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:02 smithi092 ceph-mon[29810]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:02.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:02 smithi173 ceph-mon[25952]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:02.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:46:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:46:04.013 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:46:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:03] "GET /metrics HTTP/1.1" 200 191503 "" "Prometheus/2.33.4" 2024-06-21T01:46:04.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:04 smithi173 ceph-mon[25952]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:04.769 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:04 smithi092 ceph-mon[25107]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:04.769 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:04 smithi092 ceph-mon[29810]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:05.147 INFO:teuthology.orchestra.run.smithi092.stdout:133143986201 2024-06-21T01:46:05.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-21T01:46:05.485 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:46:05.665 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:46:06.254 INFO:teuthology.orchestra.run.smithi092.stdout:55834574888 2024-06-21T01:46:06.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-21T01:46:06.501 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:06 smithi092 ceph-mon[25107]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:06.502 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:06 smithi092 ceph-mon[29810]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:06.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:06 smithi173 ceph-mon[25952]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:06.764 INFO:teuthology.orchestra.run.smithi092.stdout:201863462930 2024-06-21T01:46:06.764 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-21T01:46:07.684 INFO:teuthology.orchestra.run.smithi092.stdout:34359738414 2024-06-21T01:46:08.207 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:46:08.510 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:08 smithi092 ceph-mon[25107]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:08.510 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:08 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1615910029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-21T01:46:08.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:08 smithi092 ceph-mon[29810]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:08.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:08 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1615910029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-21T01:46:08.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:08 smithi173 ceph-mon[25952]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:08.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:08 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1615910029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-21T01:46:10.379 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738412 got 34359738414 for osd.0 2024-06-21T01:46:10.379 DEBUG:teuthology.parallel:result is None 2024-06-21T01:46:10.384 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:46:10.384 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:46:10.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:10 smithi092 ceph-mon[25107]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:10.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:10 smithi092 ceph-mon[29810]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:10.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:10 smithi173 ceph-mon[25952]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:10.805 INFO:teuthology.orchestra.run.smithi092.stdout:236223201293 2024-06-21T01:46:11.701 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:11 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3394870220' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-21T01:46:11.701 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:11 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3394870220' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-21T01:46:11.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:11 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3394870220' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-21T01:46:12.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:12 smithi173 ceph-mon[25952]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:12.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:46:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:46:12.789 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:12 smithi092 ceph-mon[25107]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:12.789 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:12 smithi092 ceph-mon[29810]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:13.647 INFO:teuthology.orchestra.run.smithi092.stdout:167503724566 2024-06-21T01:46:13.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-21T01:46:13.905 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:46:13 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:13] "GET /metrics HTTP/1.1" 200 191499 "" "Prometheus/2.33.4" 2024-06-21T01:46:14.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:14 smithi173 ceph-mon[25952]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:14.788 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:14 smithi092 ceph-mon[25107]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:14.788 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:14 smithi092 ceph-mon[29810]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:16.206 INFO:teuthology.orchestra.run.smithi092.stdout:201863462933 2024-06-21T01:46:16.684 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:46:16.686 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201291 got 236223201293 for osd.7 2024-06-21T01:46:16.687 DEBUG:teuthology.parallel:result is None 2024-06-21T01:46:16.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:16 smithi173 ceph-mon[25952]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:16.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:16 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/27296619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-21T01:46:16.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:16 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3129023220' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-21T01:46:16.767 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:16 smithi092 ceph-mon[29810]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:16.767 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:16 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/27296619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-21T01:46:16.768 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:16 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3129023220' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-21T01:46:16.768 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:16 smithi092 ceph-mon[25107]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:16.768 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:16 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/27296619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-21T01:46:16.768 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:16 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3129023220' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-21T01:46:16.786 INFO:teuthology.orchestra.run.smithi092.stdout:133143986204 2024-06-21T01:46:17.864 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462930 got 201863462933 for osd.6 2024-06-21T01:46:17.864 DEBUG:teuthology.parallel:result is None 2024-06-21T01:46:18.332 INFO:teuthology.orchestra.run.smithi092.stdout:77309411366 2024-06-21T01:46:18.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:18 smithi173 ceph-mon[25952]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:18.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:18 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/311827855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-21T01:46:18.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:18 smithi092 ceph-mon[25107]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:18.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:18 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/311827855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-21T01:46:18.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:18 smithi092 ceph-mon[29810]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:18.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:18 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/311827855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-21T01:46:18.987 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411361 got 77309411366 for osd.2 2024-06-21T01:46:18.988 DEBUG:teuthology.parallel:result is None 2024-06-21T01:46:19.368 INFO:teuthology.orchestra.run.smithi092.stdout:55834574891 2024-06-21T01:46:19.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:19 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3496400032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-21T01:46:19.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:19 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3496400032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-21T01:46:19.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:19 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3496400032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-21T01:46:19.971 INFO:teuthology.orchestra.run.smithi092.stdout:111669149730 2024-06-21T01:46:20.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:20 smithi173 ceph-mon[25952]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:20.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:20 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1479861213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-21T01:46:20.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:20 smithi092 ceph-mon[29810]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:20.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:20 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1479861213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-21T01:46:20.800 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:20 smithi092 ceph-mon[25107]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:20.800 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:20 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1479861213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-21T01:46:20.824 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986201 got 133143986204 for osd.4 2024-06-21T01:46:20.824 DEBUG:teuthology.parallel:result is None 2024-06-21T01:46:21.404 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149727 got 111669149730 for osd.3 2024-06-21T01:46:21.404 DEBUG:teuthology.parallel:result is None 2024-06-21T01:46:22.602 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574891 for osd.1 2024-06-21T01:46:22.603 DEBUG:teuthology.parallel:result is None 2024-06-21T01:46:22.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:22 smithi173 ceph-mon[25952]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:22.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:46:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:46:22.811 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:22 smithi092 ceph-mon[25107]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:22.812 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:22 smithi092 ceph-mon[29810]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:23.701 INFO:teuthology.orchestra.run.smithi092.stdout:167503724569 2024-06-21T01:46:24.049 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:23] "GET /metrics HTTP/1.1" 200 191493 "" "Prometheus/2.33.4" 2024-06-21T01:46:24.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[29810]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:24.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:24.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:24.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[29810]: Deploying daemon alertmanager.a on smithi092 2024-06-21T01:46:24.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:46:24.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:46:24.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:46:24.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:46:24.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1622972621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-21T01:46:24.050 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[25107]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:24.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:24.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:24.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[25107]: Deploying daemon alertmanager.a on smithi092 2024-06-21T01:46:24.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:46:24.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:46:24.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:46:24.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:46:24.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:23 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1622972621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-21T01:46:24.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:23 smithi173 ceph-mon[25952]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:23 smithi173 ceph-mon[25952]: Deploying daemon alertmanager.a on smithi092 2024-06-21T01:46:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:46:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:46:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:46:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:46:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:23 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1622972621' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-21T01:46:24.491 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724566 got 167503724569 for osd.5 2024-06-21T01:46:24.491 DEBUG:teuthology.parallel:result is None 2024-06-21T01:46:24.492 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-21T01:46:24.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-21T01:46:24.713 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:46:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:25 smithi092 ceph-mon[25107]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:26.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:25 smithi092 ceph-mon[29810]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:26.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:25 smithi173 ceph-mon[25952]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:26.827 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:46:26.828 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-06-21T01:46:27.841 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-06-21T01:46:26.810801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48972,"kb_used_data":6156,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912244,"statfs":{"total":767960285184,"available":767910137856,"internally_reserved":0,"allocated":6303744,"data_stored":3511211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002225"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-06-21T01:45:22.886021+0000","last_change":"2024-06-21T01:45:10.445705+0000","last_active":"2024-06-21T01:45:22.886021+0000","last_peered":"2024-06-21T01:45:22.886021+0000","last_clean":"2024-06-21T01:45:22.886021+0000","last_became_active":"2024-06-21T01:45:10.445167+0000","last_became_peered":"2024-06-21T01:45:10.445167+0000","last_unstale":"2024-06-21T01:45:22.886021+0000","last_undegraded":"2024-06-21T01:45:22.886021+0000","last_fullsized":"2024-06-21T01:45:22.886021+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-21T01:43:17.850417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-21T01:43:17.850417+0000","last_clean_scrub_stamp":"2024-06-21T01:43:17.850417+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-22T04:50:05.285730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.58199999999999996}]},{"osd":2,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.63800000000000001}]},{"osd":3,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.56999999999999995}]},{"osd":4,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.29399999999999998}]},{"osd":5,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.33500000000000002}]},{"osd":6,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.32500000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6160,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738992,"statfs":{"total":95995035648,"available":95988727808,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.43099999999999999}]},{"osd":1,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.497}]},{"osd":2,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.47299999999999998}]},{"osd":3,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.40999999999999998}]},{"osd":4,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.375}]},{"osd":5,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Jun 21 01:46:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.56200000000000006}]}]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35699999999999998,"15min":0.35499999999999998},"min":{"1min":0.245,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.50900000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.214},{"interface":"front","average":{"1min":0.37,"5min":0.34499999999999997,"15min":0.33700000000000002},"min":{"1min":0.24099999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.307}]},{"osd":2,"last update":"Fri Jun 21 01:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38,"15min":0.376},"min":{"1min":0.252,"5min":0.248,"15min":0.248},"max":{"1min":0.48999999999999999,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.315},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.28799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55700000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.28799999999999998}]},{"osd":3,"last update":"Fri Jun 21 01:45:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41499999999999998,"15min":0.41799999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40600000000000003,"15min":0.40400000000000003},"min":{"1min":0.30299999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.253}]},{"osd":4,"last update":"Fri Jun 21 01:46:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.40200000000000002}]},{"osd":5,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":12.599,"5min":12.599,"15min":12.599},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51300000000000001}]},{"osd":6,"last update":"Fri Jun 21 01:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":0.432}]},{"osd":7,"last update":"Fri Jun 21 01:46:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6924,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738228,"statfs":{"total":95995035648,"available":95987945472,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 21 01:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.34699999999999998,"15min":0.33500000000000002},"min":{"1min":0.25800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.49099999999999999,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.35799999999999998,"15min":0.34399999999999997},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.495,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.32300000000000001}]},{"osd":2,"last update":"Fri Jun 21 01:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.373,"15min":0.371},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.40200000000000002,"15min":0.40000000000000002},"min":{"1min":0.27000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.28599999999999998}]},{"osd":3,"last update":"Fri Jun 21 01:45:39 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45000000000000001,"15min":0.45200000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.42999999999999999,"15min":0.432},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.30499999999999999}]},{"osd":4,"last update":"Fri Jun 21 01:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Jun 21 01:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.61599999999999999}]},{"osd":6,"last update":"Fri Jun 21 01:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.68200000000000005}]},{"osd":7,"last update":"Fri Jun 21 01:46:09 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.503},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.59399999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36699999999999999,"15min":0.36799999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52000000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.45000000000000001}]},{"osd":1,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39900000000000002,"15min":0.40400000000000003},"min":{"1min":0.28299999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.496,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.40799999999999997,"15min":0.41199999999999998},"min":{"1min":0.30199999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55400000000000005,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.55800000000000005}]},{"osd":3,"last update":"Fri Jun 21 01:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.42999999999999999,"15min":0.44},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.47799999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.46500000000000002,"15min":0.47699999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.46600000000000003,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.52900000000000003}]},{"osd":4,"last update":"Fri Jun 21 01:46:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.69799999999999995}]},{"osd":5,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68000000000000005}]},{"osd":6,"last update":"Fri Jun 21 01:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.75800000000000001}]},{"osd":7,"last update":"Fri Jun 21 01:46:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.72499999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:45:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.40100000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39400000000000002,"15min":0.39100000000000001},"min":{"1min":0.26200000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.35199999999999998}]},{"osd":1,"last update":"Fri Jun 21 01:45:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.56799999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.315},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.27400000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Fri Jun 21 01:45:43 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59999999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.42499999999999999,"15min":0.42799999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.54800000000000004,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.33300000000000002}]},{"osd":4,"last update":"Fri Jun 21 01:46:05 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.55500000000000005,"15min":0.54300000000000004},"min":{"1min":0.40100000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.42999999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.66900000000000004}]},{"osd":5,"last update":"Fri Jun 21 01:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.41699999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.42999999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81599999999999995,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.71299999999999997}]},{"osd":6,"last update":"Fri Jun 21 01:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.68700000000000006}]},{"osd":7,"last update":"Fri Jun 21 01:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.70499999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986206,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:46:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.60499999999999998,"15min":0.61599999999999999},"min":{"1min":0.443,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71199999999999997,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.59599999999999997,"15min":0.60799999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.61899999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.67600000000000005}]},{"osd":1,"last update":"Fri Jun 21 01:46:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56999999999999995,"15min":0.57799999999999996},"min":{"1min":0.40200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.73399999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.45800000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68200000000000005,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.65600000000000003}]},{"osd":2,"last update":"Fri Jun 21 01:46:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.68899999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.498,"5min":0.57399999999999995,"15min":0.58599999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.66300000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.58999999999999997}]},{"osd":3,"last update":"Fri Jun 21 01:46:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.58799999999999997,"15min":0.60099999999999998},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.61399999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58199999999999996,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.67500000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.64100000000000001}]},{"osd":5,"last update":"Fri Jun 21 01:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.52000000000000002}]},{"osd":6,"last update":"Fri Jun 21 01:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.63300000000000001}]},{"osd":7,"last update":"Fri Jun 21 01:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53800000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724570,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.5},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53800000000000003}]},{"osd":1,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.51100000000000001}]},{"osd":2,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.496},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.54800000000000004}]},{"osd":3,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.52500000000000002}]},{"osd":4,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.318}]},{"osd":6,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.42999999999999999}]},{"osd":7,"last update":"Fri Jun 21 01:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.36799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-21T01:46:27.843 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-21T01:46:28.060 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:46:28.440 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:28 smithi092 ceph-mon[25107]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:28.441 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:28 smithi092 ceph-mon[25107]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-21T01:46:28.441 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:28 smithi092 ceph-mon[29810]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:28.441 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:28 smithi092 ceph-mon[29810]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-21T01:46:28.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:28 smithi173 ceph-mon[25952]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:28.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:28 smithi173 ceph-mon[25952]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-21T01:46:30.396 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:46:30.398 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-06-21T01:46:30.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:30 smithi092 ceph-mon[25107]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:30.574 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:30 smithi092 ceph-mon[29810]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:30.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:30 smithi173 ceph-mon[25952]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:31.530 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:31 smithi092 ceph-mon[25107]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-21T01:46:31.530 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:31 smithi092 ceph-mon[29810]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-21T01:46:31.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:31 smithi173 ceph-mon[25952]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-21T01:46:32.698 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:32 smithi092 ceph-mon[25107]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:32.698 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:32 smithi092 ceph-mon[29810]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:32.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:32 smithi173 ceph-mon[25952]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:32.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:46:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:46:32.991 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-06-21T01:46:28.811063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48972,"kb_used_data":6156,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912244,"statfs":{"total":767960285184,"available":767910137856,"internally_reserved":0,"allocated":6303744,"data_stored":3511211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002082"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-06-21T01:45:22.886021+0000","last_change":"2024-06-21T01:45:10.445705+0000","last_active":"2024-06-21T01:45:22.886021+0000","last_peered":"2024-06-21T01:45:22.886021+0000","last_clean":"2024-06-21T01:45:22.886021+0000","last_became_active":"2024-06-21T01:45:10.445167+0000","last_became_peered":"2024-06-21T01:45:10.445167+0000","last_unstale":"2024-06-21T01:45:22.886021+0000","last_undegraded":"2024-06-21T01:45:22.886021+0000","last_fullsized":"2024-06-21T01:45:22.886021+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-21T01:43:17.850417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-21T01:43:17.850417+0000","last_clean_scrub_stamp":"2024-06-21T01:43:17.850417+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-22T04:50:05.285730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.58199999999999996}]},{"osd":2,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.63800000000000001}]},{"osd":3,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.56999999999999995}]},{"osd":4,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.29399999999999998}]},{"osd":5,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.33500000000000002}]},{"osd":6,"last update":"Fri Jun 21 01:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.32500000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6160,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738992,"statfs":{"total":95995035648,"available":95988727808,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.43099999999999999}]},{"osd":1,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.497}]},{"osd":2,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.47299999999999998}]},{"osd":3,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.40999999999999998}]},{"osd":4,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.375}]},{"osd":5,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Jun 21 01:46:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.56200000000000006}]}]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:45:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35699999999999998,"15min":0.35499999999999998},"min":{"1min":0.245,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.50900000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.214},{"interface":"front","average":{"1min":0.37,"5min":0.34499999999999997,"15min":0.33700000000000002},"min":{"1min":0.24099999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.307}]},{"osd":2,"last update":"Fri Jun 21 01:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38,"15min":0.376},"min":{"1min":0.252,"5min":0.248,"15min":0.248},"max":{"1min":0.48999999999999999,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.315},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.28799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55700000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.28799999999999998}]},{"osd":3,"last update":"Fri Jun 21 01:45:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41499999999999998,"15min":0.41799999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40600000000000003,"15min":0.40400000000000003},"min":{"1min":0.30299999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.253}]},{"osd":4,"last update":"Fri Jun 21 01:46:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.40200000000000002}]},{"osd":5,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":12.599,"5min":12.599,"15min":12.599},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51300000000000001}]},{"osd":6,"last update":"Fri Jun 21 01:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":0.432}]},{"osd":7,"last update":"Fri Jun 21 01:46:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6924,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738228,"statfs":{"total":95995035648,"available":95987945472,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jun 21 01:45:58 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.34699999999999998,"15min":0.33500000000000002},"min":{"1min":0.25800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.49099999999999999,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.35799999999999998,"15min":0.34399999999999997},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.495,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.32300000000000001}]},{"osd":2,"last update":"Fri Jun 21 01:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.373,"15min":0.371},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.40200000000000002,"15min":0.40000000000000002},"min":{"1min":0.27000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.28599999999999998}]},{"osd":3,"last update":"Fri Jun 21 01:45:39 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45000000000000001,"15min":0.45200000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.42999999999999999,"15min":0.432},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.30499999999999999}]},{"osd":4,"last update":"Fri Jun 21 01:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.54700000000000004,"15min":0.54300000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Jun 21 01:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.61599999999999999}]},{"osd":6,"last update":"Fri Jun 21 01:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.68200000000000005}]},{"osd":7,"last update":"Fri Jun 21 01:46:09 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.503},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.59399999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36699999999999999,"15min":0.36799999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52000000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.45000000000000001}]},{"osd":1,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.39900000000000002,"15min":0.40400000000000003},"min":{"1min":0.28299999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.496,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38,"5min":0.40799999999999997,"15min":0.41199999999999998},"min":{"1min":0.30199999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55400000000000005,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.55800000000000005}]},{"osd":3,"last update":"Fri Jun 21 01:45:41 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.42999999999999999,"15min":0.44},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.47799999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.46500000000000002,"15min":0.47699999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.46600000000000003,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.52900000000000003}]},{"osd":4,"last update":"Fri Jun 21 01:46:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.69799999999999995}]},{"osd":5,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68000000000000005}]},{"osd":6,"last update":"Fri Jun 21 01:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.75800000000000001}]},{"osd":7,"last update":"Fri Jun 21 01:46:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.72499999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:45:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.40500000000000003,"15min":0.40100000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39400000000000002,"15min":0.39100000000000001},"min":{"1min":0.26200000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.35199999999999998}]},{"osd":1,"last update":"Fri Jun 21 01:45:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.56799999999999995,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.315},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.27400000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Fri Jun 21 01:45:43 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.59999999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.42499999999999999,"15min":0.42799999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.54800000000000004,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.33300000000000002}]},{"osd":4,"last update":"Fri Jun 21 01:46:05 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.55500000000000005,"15min":0.54300000000000004},"min":{"1min":0.40100000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.42999999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.66900000000000004}]},{"osd":5,"last update":"Fri Jun 21 01:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.41699999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60299999999999998,"15min":0.60099999999999998},"min":{"1min":0.42999999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81599999999999995,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.71299999999999997}]},{"osd":6,"last update":"Fri Jun 21 01:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.68700000000000006}]},{"osd":7,"last update":"Fri Jun 21 01:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.70499999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986206,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:46:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.60499999999999998,"15min":0.61599999999999999},"min":{"1min":0.443,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71199999999999997,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.59599999999999997,"15min":0.60799999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.61899999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.67600000000000005}]},{"osd":1,"last update":"Fri Jun 21 01:46:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56999999999999995,"15min":0.57799999999999996},"min":{"1min":0.40200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.73399999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.45800000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68200000000000005,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.65600000000000003}]},{"osd":2,"last update":"Fri Jun 21 01:46:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.68899999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.498,"5min":0.57399999999999995,"15min":0.58599999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.66300000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.58999999999999997}]},{"osd":3,"last update":"Fri Jun 21 01:46:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.58799999999999997,"15min":0.60099999999999998},"min":{"1min":0.40899999999999997,"5min":0.38,"15min":0.38},"max":{"1min":0.61399999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58199999999999996,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.67500000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.64100000000000001}]},{"osd":5,"last update":"Fri Jun 21 01:45:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.52000000000000002}]},{"osd":6,"last update":"Fri Jun 21 01:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.63300000000000001}]},{"osd":7,"last update":"Fri Jun 21 01:46:10 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53800000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724570,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.5},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53800000000000003}]},{"osd":1,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.51100000000000001}]},{"osd":2,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.496},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.54800000000000004}]},{"osd":3,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.52500000000000002}]},{"osd":4,"last update":"Fri Jun 21 01:45:21 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.318}]},{"osd":6,"last update":"Fri Jun 21 01:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.42999999999999999}]},{"osd":7,"last update":"Fri Jun 21 01:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.36799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-21T01:46:32.993 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-21T01:46:32.993 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-21T01:46:32.993 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-21T01:46:32.993 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-21T01:46:33.746 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/mon.a/config 2024-06-21T01:46:33.841 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:46:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:33] "GET /metrics HTTP/1.1" 200 191493 "" "Prometheus/2.33.4" 2024-06-21T01:46:34.566 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:34 smithi092 ceph-mon[25107]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:34.566 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:34 smithi092 ceph-mon[29810]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:34.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:34 smithi173 ceph-mon[25952]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:35.611 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-06-21T01:46:35.612 INFO:teuthology.orchestra.run.smithi092.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-21T01:46:36.405 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-06-21T01:46:36.405 INFO:tasks.cephadm:Setup complete, yielding 2024-06-21T01:46:36.405 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-21T01:46:36.414 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi092.front.sepia.ceph.com 2024-06-21T01:46:36.414 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-06-21T01:46:36.660 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:36 smithi092 ceph-mon[25107]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:36.660 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:36 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2101324574' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-21T01:46:36.660 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:36 smithi092 ceph-mon[29810]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:36.660 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:36 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2101324574' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-21T01:46:36.661 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:46:36 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[48084]: level=info ts=2024-06-21T01:46:36.624Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-06-21T01:46:36.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:36 smithi173 ceph-mon[25952]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:36.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:36 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2101324574' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-21T01:46:36.953 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:46:36 smithi092 bash[48056]: e7db2ea37246f7729883f8bf80e746fdbbd835bbead7c95e57ce86ebe3a52519 2024-06-21T01:46:36.953 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:46:36 smithi092 systemd[1]: Started Ceph alertmanager.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:46:37.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:37 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:37.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:37 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:37.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:37 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:37.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:37 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:46:37.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:37 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:37.981 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:37 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:37.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:37 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:37.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:37 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:37.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:37 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:46:37.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:37 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:38.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:37 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:38.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:37 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:38.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:37 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:38.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:37 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:46:38.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:37 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:38.747 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:46:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[48084]: level=info ts=2024-06-21T01:46:38.591Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000827149s 2024-06-21T01:46:38.914 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-21T01:46:38.926 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi092.front.sepia.ceph.com 2024-06-21T01:46:38.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-06-21T01:46:39.013 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:38 smithi092 ceph-mon[29810]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:39.013 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:38 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:46:39.013 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:38 smithi092 ceph-mon[29810]: Deploying daemon grafana.a on smithi173 2024-06-21T01:46:39.013 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:38 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:39.013 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:38 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3526286609' entity='client.admin' 2024-06-21T01:46:39.013 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:38 smithi092 ceph-mon[25107]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:39.013 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:38 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:46:39.013 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:38 smithi092 ceph-mon[25107]: Deploying daemon grafana.a on smithi173 2024-06-21T01:46:39.014 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:38 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:39.014 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:38 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3526286609' entity='client.admin' 2024-06-21T01:46:39.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:38 smithi173 ceph-mon[25952]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:39.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:38 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:46:39.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:38 smithi173 ceph-mon[25952]: Deploying daemon grafana.a on smithi173 2024-06-21T01:46:39.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:38 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:39.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:38 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3526286609' entity='client.admin' 2024-06-21T01:46:41.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:40 smithi092 ceph-mon[29810]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:41.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:40 smithi092 ceph-mon[25107]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:41.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:40 smithi173 ceph-mon[25952]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:41.915 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:46:41.915 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "f0ab9d48-c21e-49db-8247-fc624db606e0", 2024-06-21T01:46:41.915 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "r", 2024-06-21T01:46:41.915 INFO:teuthology.orchestra.run.smithi092.stdout: "current_period": "da160ac5-08ce-45ad-8d6a-7b1355a956e9", 2024-06-21T01:46:41.915 INFO:teuthology.orchestra.run.smithi092.stdout: "epoch": 1 2024-06-21T01:46:41.916 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:46:42.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:41 smithi092 ceph-mon[25107]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:42.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:41 smithi092 ceph-mon[25107]: osdmap e61: 8 total, 8 up, 8 in 2024-06-21T01:46:42.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:41 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2802712543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-21T01:46:42.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:41 smithi092 ceph-mon[25107]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-21T01:46:42.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:41 smithi092 ceph-mon[29810]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:42.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:41 smithi092 ceph-mon[29810]: osdmap e61: 8 total, 8 up, 8 in 2024-06-21T01:46:42.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:41 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2802712543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-21T01:46:42.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:41 smithi092 ceph-mon[29810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-21T01:46:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:41 smithi173 ceph-mon[25952]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:41 smithi173 ceph-mon[25952]: osdmap e61: 8 total, 8 up, 8 in 2024-06-21T01:46:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:41 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2802712543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-21T01:46:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:41 smithi173 ceph-mon[25952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-21T01:46:42.491 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-06-21T01:46:42.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:46:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:46:43.057 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:42 smithi092 ceph-mon[25107]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-06-21T01:46:43.057 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:42 smithi092 ceph-mon[25107]: osdmap e62: 8 total, 8 up, 8 in 2024-06-21T01:46:43.057 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:42 smithi092 ceph-mon[25107]: osdmap e63: 8 total, 8 up, 8 in 2024-06-21T01:46:43.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:42 smithi092 ceph-mon[29810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-06-21T01:46:43.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:42 smithi092 ceph-mon[29810]: osdmap e62: 8 total, 8 up, 8 in 2024-06-21T01:46:43.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:42 smithi092 ceph-mon[29810]: osdmap e63: 8 total, 8 up, 8 in 2024-06-21T01:46:43.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:42 smithi173 ceph-mon[25952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-06-21T01:46:43.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:42 smithi173 ceph-mon[25952]: osdmap e62: 8 total, 8 up, 8 in 2024-06-21T01:46:43.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:42 smithi173 ceph-mon[25952]: osdmap e63: 8 total, 8 up, 8 in 2024-06-21T01:46:43.746 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:46:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:43] "GET /metrics HTTP/1.1" 200 191505 "" "Prometheus/2.33.4" 2024-06-21T01:46:44.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:43 smithi092 ceph-mon[29810]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:44.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:43 smithi092 ceph-mon[25107]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:44.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:43 smithi173 ceph-mon[25952]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:46:44.294 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:46:44.294 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "0b23d956-ce2f-4e06-8e5c-84411e927b1f", 2024-06-21T01:46:44.294 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "default", 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "api_name": "default", 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "is_master": "true", 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "endpoints": [], 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "hostnames": [], 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "hostnames_s3website": [], 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "master_zone": "", 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "zones": [], 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "placement_targets": [], 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "default_placement": "", 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_id": "f0ab9d48-c21e-49db-8247-fc624db606e0", 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "sync_policy": { 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: "groups": [] 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:46:44.295 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:46:44.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-06-21T01:46:46.222 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:45 smithi092 ceph-mon[25107]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-06-21T01:46:46.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:45 smithi092 ceph-mon[29810]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-06-21T01:46:46.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:45 smithi173 ceph-mon[25952]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-06-21T01:46:46.534 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "2fa38846-d7cc-47cf-b421-e83118206527", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "z", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "domain_root": "z.rgw.meta:root", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "control_pool": "z.rgw.control", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "gc_pool": "z.rgw.log:gc", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "lc_pool": "z.rgw.log:lc", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "log_pool": "z.rgw.log", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "roles_pool": "z.rgw.meta:roles", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-06-21T01:46:46.535 INFO:teuthology.orchestra.run.smithi092.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "otp_pool": "z.rgw.otp", 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "system_key": { 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "access_key": "", 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "secret_key": "" 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "placement_pools": [ 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "key": "default-placement", 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "val": { 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "index_pool": "z.rgw.buckets.index", 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "storage_classes": { 2024-06-21T01:46:46.536 INFO:teuthology.orchestra.run.smithi092.stdout: "STANDARD": { 2024-06-21T01:46:46.537 INFO:teuthology.orchestra.run.smithi092.stdout: "data_pool": "z.rgw.buckets.data" 2024-06-21T01:46:46.537 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:46:46.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:46:46.537 INFO:teuthology.orchestra.run.smithi092.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-06-21T01:46:46.537 INFO:teuthology.orchestra.run.smithi092.stdout: "index_type": 0 2024-06-21T01:46:46.537 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:46:46.537 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:46:46.537 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:46:46.537 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_id": "f0ab9d48-c21e-49db-8247-fc624db606e0", 2024-06-21T01:46:46.537 INFO:teuthology.orchestra.run.smithi092.stdout: "notif_pool": "z.rgw.log:notif" 2024-06-21T01:46:46.537 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:46:46.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:46:46 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[48084]: level=info ts=2024-06-21T01:46:46.593Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002914339s 2024-06-21T01:46:47.137 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-06-21T01:46:48.531 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:48 smithi092 ceph-mon[29810]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-06-21T01:46:48.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:48 smithi092 ceph-mon[25107]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-06-21T01:46:48.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:48 smithi173 ceph-mon[25952]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-06-21T01:46:50.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:50 smithi173 ceph-mon[25952]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s wr, 1 op/s 2024-06-21T01:46:50.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:50 smithi173 ceph-mon[25952]: osdmap e64: 8 total, 8 up, 8 in 2024-06-21T01:46:50.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:50 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-06-21T01:46:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:50 smithi092 ceph-mon[25107]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s wr, 1 op/s 2024-06-21T01:46:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:50 smithi092 ceph-mon[25107]: osdmap e64: 8 total, 8 up, 8 in 2024-06-21T01:46:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:50 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-06-21T01:46:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:50 smithi092 ceph-mon[29810]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s wr, 1 op/s 2024-06-21T01:46:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:50 smithi092 ceph-mon[29810]: osdmap e64: 8 total, 8 up, 8 in 2024-06-21T01:46:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:50 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-06-21T01:46:51.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:51 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-06-21T01:46:51.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:51 smithi173 ceph-mon[25952]: osdmap e65: 8 total, 8 up, 8 in 2024-06-21T01:46:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:51 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-06-21T01:46:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:51 smithi092 ceph-mon[25107]: osdmap e65: 8 total, 8 up, 8 in 2024-06-21T01:46:51.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:51 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-06-21T01:46:51.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:51 smithi092 ceph-mon[29810]: osdmap e65: 8 total, 8 up, 8 in 2024-06-21T01:46:52.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:52 smithi173 ceph-mon[25952]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-06-21T01:46:52.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:52 smithi173 ceph-mon[25952]: osdmap e66: 8 total, 8 up, 8 in 2024-06-21T01:46:52.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:52 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-06-21T01:46:52.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:46:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:46:52.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:52 smithi092 ceph-mon[25107]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-06-21T01:46:52.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:52 smithi092 ceph-mon[25107]: osdmap e66: 8 total, 8 up, 8 in 2024-06-21T01:46:52.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:52 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-06-21T01:46:52.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:52 smithi092 ceph-mon[29810]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-06-21T01:46:52.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:52 smithi092 ceph-mon[29810]: osdmap e66: 8 total, 8 up, 8 in 2024-06-21T01:46:52.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:52 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-06-21T01:46:53.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:53 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-06-21T01:46:53.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:53 smithi092 ceph-mon[25107]: osdmap e67: 8 total, 8 up, 8 in 2024-06-21T01:46:53.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:53 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:53.734 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:53 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-06-21T01:46:53.734 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:53 smithi092 ceph-mon[29810]: osdmap e67: 8 total, 8 up, 8 in 2024-06-21T01:46:53.734 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:53 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:53.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:53 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3473039791' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-06-21T01:46:53.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:53 smithi173 ceph-mon[25952]: osdmap e67: 8 total, 8 up, 8 in 2024-06-21T01:46:53.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:53 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:46:54.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:46:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:46:53] "GET /metrics HTTP/1.1" 200 195328 "" "Prometheus/2.33.4" 2024-06-21T01:46:54.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:54 smithi173 ceph-mon[25952]: pgmap v55: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-06-21T01:46:54.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:54 smithi173 ceph-mon[25952]: osdmap e68: 8 total, 8 up, 8 in 2024-06-21T01:46:54.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:54 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-06-21T01:46:54.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:54 smithi092 ceph-mon[25107]: pgmap v55: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-06-21T01:46:54.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:54 smithi092 ceph-mon[25107]: osdmap e68: 8 total, 8 up, 8 in 2024-06-21T01:46:54.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:54 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-06-21T01:46:54.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:54 smithi092 ceph-mon[29810]: pgmap v55: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-06-21T01:46:54.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:54 smithi092 ceph-mon[29810]: osdmap e68: 8 total, 8 up, 8 in 2024-06-21T01:46:54.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:54 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-06-21T01:46:55.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:55 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-06-21T01:46:55.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:55 smithi092 ceph-mon[25107]: osdmap e69: 8 total, 8 up, 8 in 2024-06-21T01:46:55.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:55 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-06-21T01:46:55.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:55 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-06-21T01:46:55.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:55 smithi092 ceph-mon[29810]: osdmap e69: 8 total, 8 up, 8 in 2024-06-21T01:46:55.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:55 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-06-21T01:46:55.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:55 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-06-21T01:46:55.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:55 smithi173 ceph-mon[25952]: osdmap e69: 8 total, 8 up, 8 in 2024-06-21T01:46:55.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:55 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-06-21T01:46:56.540 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:46:56.540 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "0bab827f-3cd0-4766-938f-b2ed41293dff", 2024-06-21T01:46:56.540 INFO:teuthology.orchestra.run.smithi092.stdout: "epoch": 1, 2024-06-21T01:46:56.540 INFO:teuthology.orchestra.run.smithi092.stdout: "predecessor_uuid": "da160ac5-08ce-45ad-8d6a-7b1355a956e9", 2024-06-21T01:46:56.540 INFO:teuthology.orchestra.run.smithi092.stdout: "sync_status": [], 2024-06-21T01:46:56.540 INFO:teuthology.orchestra.run.smithi092.stdout: "period_map": { 2024-06-21T01:46:56.540 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "0bab827f-3cd0-4766-938f-b2ed41293dff", 2024-06-21T01:46:56.540 INFO:teuthology.orchestra.run.smithi092.stdout: "zonegroups": [ 2024-06-21T01:46:56.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-06-21T01:46:56.540 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "0b23d956-ce2f-4e06-8e5c-84411e927b1f", 2024-06-21T01:46:56.540 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "default", 2024-06-21T01:46:56.541 INFO:teuthology.orchestra.run.smithi092.stdout: "api_name": "default", 2024-06-21T01:46:56.541 INFO:teuthology.orchestra.run.smithi092.stdout: "is_master": "true", 2024-06-21T01:46:56.541 INFO:teuthology.orchestra.run.smithi092.stdout: "endpoints": [], 2024-06-21T01:46:56.541 INFO:teuthology.orchestra.run.smithi092.stdout: "hostnames": [], 2024-06-21T01:46:56.541 INFO:teuthology.orchestra.run.smithi092.stdout: "hostnames_s3website": [], 2024-06-21T01:46:56.541 INFO:teuthology.orchestra.run.smithi092.stdout: "master_zone": "2fa38846-d7cc-47cf-b421-e83118206527", 2024-06-21T01:46:56.541 INFO:teuthology.orchestra.run.smithi092.stdout: "zones": [ 2024-06-21T01:46:56.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-06-21T01:46:56.541 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "2fa38846-d7cc-47cf-b421-e83118206527", 2024-06-21T01:46:56.541 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "z", 2024-06-21T01:46:56.541 INFO:teuthology.orchestra.run.smithi092.stdout: "endpoints": [], 2024-06-21T01:46:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "log_meta": "false", 2024-06-21T01:46:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "log_data": "false", 2024-06-21T01:46:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "bucket_index_max_shards": 11, 2024-06-21T01:46:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "read_only": "false", 2024-06-21T01:46:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_type": "", 2024-06-21T01:46:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "sync_from_all": "true", 2024-06-21T01:46:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "sync_from": [], 2024-06-21T01:46:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "redirect_zone": "" 2024-06-21T01:46:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:46:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: "placement_targets": [ 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "default-placement", 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: "storage_classes": [ 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: "STANDARD" 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: "default_placement": "default-placement", 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_id": "f0ab9d48-c21e-49db-8247-fc624db606e0", 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: "sync_policy": { 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: "groups": [] 2024-06-21T01:46:56.543 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: "short_zone_ids": [ 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: "key": "2fa38846-d7cc-47cf-b421-e83118206527", 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 124736998 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: "master_zonegroup": "0b23d956-ce2f-4e06-8e5c-84411e927b1f", 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: "master_zone": "2fa38846-d7cc-47cf-b421-e83118206527", 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: "period_config": { 2024-06-21T01:46:56.544 INFO:teuthology.orchestra.run.smithi092.stdout: "bucket_quota": { 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: "enabled": false, 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: "check_on_raw": false, 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: "max_size": -1, 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: "max_size_kb": 0, 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: "max_objects": -1 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: "user_quota": { 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: "enabled": false, 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: "check_on_raw": false, 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: "max_size": -1, 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: "max_size_kb": 0, 2024-06-21T01:46:56.545 INFO:teuthology.orchestra.run.smithi092.stdout: "max_objects": -1 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: "user_ratelimit": { 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_ops": 0, 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_ops": 0, 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_bytes": 0, 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_bytes": 0, 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: "enabled": false 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: "bucket_ratelimit": { 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_ops": 0, 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_ops": 0, 2024-06-21T01:46:56.546 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_bytes": 0, 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_bytes": 0, 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: "enabled": false 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: "anonymous_ratelimit": { 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_ops": 0, 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_ops": 0, 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_bytes": 0, 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_bytes": 0, 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: "enabled": false 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_id": "f0ab9d48-c21e-49db-8247-fc624db606e0", 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_name": "r", 2024-06-21T01:46:56.547 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_epoch": 2 2024-06-21T01:46:56.548 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:46:56.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:56 smithi092 ceph-mon[25107]: pgmap v58: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-06-21T01:46:56.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:56 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-06-21T01:46:56.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:56 smithi092 ceph-mon[25107]: osdmap e70: 8 total, 8 up, 8 in 2024-06-21T01:46:56.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:56 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-06-21T01:46:56.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:56 smithi092 ceph-mon[29810]: pgmap v58: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-06-21T01:46:56.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:56 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-06-21T01:46:56.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:56 smithi092 ceph-mon[29810]: osdmap e70: 8 total, 8 up, 8 in 2024-06-21T01:46:56.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:56 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-06-21T01:46:56.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:56 smithi173 ceph-mon[25952]: pgmap v58: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-06-21T01:46:56.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:56 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-06-21T01:46:56.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:56 smithi173 ceph-mon[25952]: osdmap e70: 8 total, 8 up, 8 in 2024-06-21T01:46:56.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:56 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-06-21T01:46:57.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-06-21T01:46:57.761 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:57 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-06-21T01:46:57.761 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:57 smithi092 ceph-mon[25107]: osdmap e71: 8 total, 8 up, 8 in 2024-06-21T01:46:57.762 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:57 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-06-21T01:46:57.762 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:57 smithi092 ceph-mon[29810]: osdmap e71: 8 total, 8 up, 8 in 2024-06-21T01:46:57.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:57 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/140796348' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-06-21T01:46:57.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:57 smithi173 ceph-mon[25952]: osdmap e71: 8 total, 8 up, 8 in 2024-06-21T01:46:58.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:46:58 smithi092 ceph-mon[25107]: pgmap v61: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-06-21T01:46:58.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:46:58 smithi092 ceph-mon[29810]: pgmap v61: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-06-21T01:46:58.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:46:58 smithi173 ceph-mon[25952]: pgmap v61: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-06-21T01:46:59.430 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled rgw.foo update... 2024-06-21T01:47:00.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph orch apply rgw smpl' 2024-06-21T01:47:00.677 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:00 smithi092 ceph-mon[25107]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 191 B/s rd, 383 B/s wr, 1 op/s 2024-06-21T01:47:00.677 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:00 smithi092 ceph-mon[25107]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:47:00.677 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:00 smithi092 ceph-mon[25107]: Saving service rgw.foo spec with placement count:2 2024-06-21T01:47:00.677 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:00 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:00.677 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:00 smithi092 ceph-mon[29810]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 191 B/s rd, 383 B/s wr, 1 op/s 2024-06-21T01:47:00.677 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:00 smithi092 ceph-mon[29810]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:47:00.677 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:00 smithi092 ceph-mon[29810]: Saving service rgw.foo spec with placement count:2 2024-06-21T01:47:00.677 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:00 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:00.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:00 smithi173 ceph-mon[25952]: pgmap v62: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 191 B/s rd, 383 B/s wr, 1 op/s 2024-06-21T01:47:00.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:00 smithi173 ceph-mon[25952]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:47:00.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:00 smithi173 ceph-mon[25952]: Saving service rgw.foo spec with placement count:2 2024-06-21T01:47:00.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:00 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:01.841 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled rgw.smpl update... 2024-06-21T01:47:02.340 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph osd pool create foo' 2024-06-21T01:47:02.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:02 smithi173 ceph-mon[25952]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-06-21T01:47:02.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:02 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:02.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:47:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:47:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:47:02.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:02 smithi092 ceph-mon[25107]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-06-21T01:47:02.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:02 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:02.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:02 smithi092 ceph-mon[29810]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-06-21T01:47:02.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:02 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:03.673 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:03 smithi092 ceph-mon[25107]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:47:03.673 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:03 smithi092 ceph-mon[25107]: Saving service rgw.smpl spec with placement count:2 2024-06-21T01:47:03.674 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:03 smithi092 ceph-mon[29810]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:47:03.674 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:03 smithi092 ceph-mon[29810]: Saving service rgw.smpl spec with placement count:2 2024-06-21T01:47:03.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:03 smithi173 ceph-mon[25952]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:47:03.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:03 smithi173 ceph-mon[25952]: Saving service rgw.smpl spec with placement count:2 2024-06-21T01:47:03.924 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:47:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:47:03] "GET /metrics HTTP/1.1" 200 195328 "" "Prometheus/2.33.4" 2024-06-21T01:47:04.716 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'foo' created 2024-06-21T01:47:04.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:04 smithi173 ceph-mon[25952]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-06-21T01:47:05.063 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:04 smithi092 ceph-mon[25107]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-06-21T01:47:05.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:04 smithi092 ceph-mon[29810]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-06-21T01:47:05.447 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'rbd pool init foo' 2024-06-21T01:47:05.752 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:05 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3727982024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-06-21T01:47:05.752 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:05 smithi092 ceph-mon[25107]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-06-21T01:47:05.752 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:05 smithi092 ceph-mon[25107]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-06-21T01:47:05.752 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:05 smithi092 ceph-mon[25107]: osdmap e72: 8 total, 8 up, 8 in 2024-06-21T01:47:05.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:05 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3727982024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-06-21T01:47:05.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:05 smithi092 ceph-mon[29810]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-06-21T01:47:05.753 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:05 smithi092 ceph-mon[29810]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-06-21T01:47:05.753 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:05 smithi092 ceph-mon[29810]: osdmap e72: 8 total, 8 up, 8 in 2024-06-21T01:47:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:05 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3727982024' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-06-21T01:47:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:05 smithi173 ceph-mon[25952]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-06-21T01:47:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:05 smithi173 ceph-mon[25952]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-06-21T01:47:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:05 smithi173 ceph-mon[25952]: osdmap e72: 8 total, 8 up, 8 in 2024-06-21T01:47:06.900 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:06 smithi173 ceph-mon[25952]: pgmap v66: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 861 B/s wr, 2 op/s 2024-06-21T01:47:06.900 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:06 smithi173 ceph-mon[25952]: osdmap e73: 8 total, 8 up, 8 in 2024-06-21T01:47:07.006 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:06 smithi092 ceph-mon[25107]: pgmap v66: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 861 B/s wr, 2 op/s 2024-06-21T01:47:07.006 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:06 smithi092 ceph-mon[25107]: osdmap e73: 8 total, 8 up, 8 in 2024-06-21T01:47:07.006 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:06 smithi092 ceph-mon[29810]: pgmap v66: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 861 B/s wr, 2 op/s 2024-06-21T01:47:07.006 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:06 smithi092 ceph-mon[29810]: osdmap e73: 8 total, 8 up, 8 in 2024-06-21T01:47:07.151 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-06-21T01:47:07.151 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-06-21T01:47:07.403 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-06-21T01:47:07.403 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-06-21T01:47:07.404 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-06-21T01:47:07.405 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-06-21T01:47:07.405 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-06-21T01:47:07.405 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-06-21T01:47:07.405 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-06-21T01:47:07.406 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-06-21T01:47:07.406 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-06-21T01:47:07.406 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-06-21T01:47:07.406 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-06-21T01:47:07.406 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-06-21T01:47:07.406 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-06-21T01:47:07.406 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-06-21T01:47:07.407 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-06-21T01:47:07.408 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-06-21T01:47:07.409 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-06-21T01:47:07.410 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-06-21T01:47:07.411 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-06-21T01:47:07.412 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-06-21T01:47:07.413 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-06-21T01:47:07.414 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-06-21T01:47:07.415 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-06-21T01:47:07.415 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-06-21T01:47:07.415 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-06-21T01:47:07.415 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-06-21T01:47:07.415 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-06-21T01:47:07.415 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-06-21T01:47:07.662 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-06-21T01:47:07.662 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-06-21T01:47:07.662 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-06-21T01:47:07.662 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-06-21T01:47:07.662 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-06-21T01:47:07.662 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-06-21T01:47:07.662 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-06-21T01:47:07.662 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-06-21T01:47:07.662 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-06-21T01:47:07.663 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-06-21T01:47:07.663 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-06-21T01:47:07.663 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-06-21T01:47:07.663 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-06-21T01:47:07.663 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-06-21T01:47:07.663 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-06-21T01:47:07.663 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-06-21T01:47:07.663 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-06-21T01:47:07.663 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-06-21T01:47:07.663 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-06-21T01:47:07.664 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-06-21T01:47:07.665 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-06-21T01:47:07.666 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-06-21T01:47:07.667 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-06-21T01:47:07.668 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-06-21T01:47:07.668 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-06-21T01:47:07.668 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-06-21T01:47:07.668 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-06-21T01:47:07.668 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-06-21T01:47:07.668 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-06-21T01:47:07.669 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-06-21T01:47:07.670 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-06-21T01:47:07.671 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=472.786503ms 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-06-21T01:47:07.672 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-06-21T01:47:07.673 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-06-21T01:47:07.673 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-06-21T01:47:07.673 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-06-21T01:47:07.673 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:47:07+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-06-21T01:47:07.932 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-mon[25952]: osdmap e74: 8 total, 8 up, 8 in 2024-06-21T01:47:07.932 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2649420560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-06-21T01:47:07.932 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:07.932 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:07.932 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:07 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:08.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:07 smithi092 ceph-mon[25107]: osdmap e74: 8 total, 8 up, 8 in 2024-06-21T01:47:08.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:07 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2649420560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-06-21T01:47:08.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:07 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:08.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:07 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:08.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:07 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:08.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:07 smithi092 ceph-mon[29810]: osdmap e74: 8 total, 8 up, 8 in 2024-06-21T01:47:08.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:07 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2649420560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-06-21T01:47:08.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:07 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:08.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:07 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:08.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:07 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:08.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:08 smithi173 ceph-mon[25952]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:47:08.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:08 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2649420560' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-06-21T01:47:08.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:08 smithi173 ceph-mon[25952]: osdmap e75: 8 total, 8 up, 8 in 2024-06-21T01:47:08.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:08 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:09.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:08 smithi092 ceph-mon[29810]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:47:09.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:08 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2649420560' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-06-21T01:47:09.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:08 smithi092 ceph-mon[29810]: osdmap e75: 8 total, 8 up, 8 in 2024-06-21T01:47:09.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:08 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:09.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:08 smithi092 ceph-mon[25107]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:47:09.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:08 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2649420560' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-06-21T01:47:09.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:08 smithi092 ceph-mon[25107]: osdmap e75: 8 total, 8 up, 8 in 2024-06-21T01:47:09.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:08 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:10.011 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:09 smithi092 ceph-mon[25107]: osdmap e76: 8 total, 8 up, 8 in 2024-06-21T01:47:10.011 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:09 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:10.011 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:09 smithi092 ceph-mon[29810]: osdmap e76: 8 total, 8 up, 8 in 2024-06-21T01:47:10.012 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:09 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:10.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:09 smithi173 ceph-mon[25952]: osdmap e76: 8 total, 8 up, 8 in 2024-06-21T01:47:10.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:09 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:10.402 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph orch apply iscsi foo u p' 2024-06-21T01:47:11.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:10 smithi092 ceph-mon[25107]: pgmap v72: 161 pgs: 27 unknown, 134 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:47:11.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:10 smithi092 ceph-mon[25107]: osdmap e77: 8 total, 8 up, 8 in 2024-06-21T01:47:11.031 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:10 smithi092 ceph-mon[29810]: pgmap v72: 161 pgs: 27 unknown, 134 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:47:11.031 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:10 smithi092 ceph-mon[29810]: osdmap e77: 8 total, 8 up, 8 in 2024-06-21T01:47:11.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:10 smithi173 ceph-mon[25952]: pgmap v72: 161 pgs: 27 unknown, 134 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:47:11.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:10 smithi173 ceph-mon[25952]: osdmap e77: 8 total, 8 up, 8 in 2024-06-21T01:47:12.309 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled iscsi.foo update... 2024-06-21T01:47:12.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:47:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:47:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:47:12.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:12 smithi092 ceph-mon[25107]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 249 B/s wr, 0 op/s 2024-06-21T01:47:12.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:12 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:12.840 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:12 smithi092 ceph-mon[29810]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 249 B/s wr, 0 op/s 2024-06-21T01:47:12.840 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:12 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:12.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'sleep 120' 2024-06-21T01:47:13.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:12 smithi173 ceph-mon[25952]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 249 B/s wr, 0 op/s 2024-06-21T01:47:13.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:12 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:13 smithi092 ceph-mon[25107]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:47:14.134 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:13 smithi092 ceph-mon[25107]: Saving service iscsi.foo spec with placement count:1 2024-06-21T01:47:14.134 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:47:13 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:47:13] "GET /metrics HTTP/1.1" 200 197864 "" "Prometheus/2.33.4" 2024-06-21T01:47:14.135 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:13 smithi092 ceph-mon[29810]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:47:14.135 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:13 smithi092 ceph-mon[29810]: Saving service iscsi.foo spec with placement count:1 2024-06-21T01:47:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:13 smithi173 ceph-mon[25952]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:47:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:13 smithi173 ceph-mon[25952]: Saving service iscsi.foo spec with placement count:1 2024-06-21T01:47:15.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:14 smithi092 ceph-mon[25107]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-06-21T01:47:15.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:14 smithi092 ceph-mon[29810]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-06-21T01:47:15.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:14 smithi173 ceph-mon[25952]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-06-21T01:47:16.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:16 smithi173 ceph-mon[25952]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 288 B/s wr, 0 op/s 2024-06-21T01:47:16.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:16 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:16.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:16 smithi173 ceph-mon[25952]: osdmap e78: 8 total, 8 up, 8 in 2024-06-21T01:47:16.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:16 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:16.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:16 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:16.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:16 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:16.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:16 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:16.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:16 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:16.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:16 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:16.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:16 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:17.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[25107]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 288 B/s wr, 0 op/s 2024-06-21T01:47:17.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:17.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[25107]: osdmap e78: 8 total, 8 up, 8 in 2024-06-21T01:47:17.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:17.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:17.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:17.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:17.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:17.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:17.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:17.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[29810]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 288 B/s wr, 0 op/s 2024-06-21T01:47:17.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:17.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[29810]: osdmap e78: 8 total, 8 up, 8 in 2024-06-21T01:47:17.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:17.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:17.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:17.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:17.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:17.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:17.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:16 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:18.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:17 smithi092 ceph-mon[29810]: Saving service rgw.foo spec with placement count:2 2024-06-21T01:47:18.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:17 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:18.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:17 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:18.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:17 smithi092 ceph-mon[29810]: Deploying daemon rgw.foo.smithi092.lzbkvh on smithi092 2024-06-21T01:47:18.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:17 smithi092 ceph-mon[25107]: Saving service rgw.foo spec with placement count:2 2024-06-21T01:47:18.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:17 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:18.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:17 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:18.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:17 smithi092 ceph-mon[25107]: Deploying daemon rgw.foo.smithi092.lzbkvh on smithi092 2024-06-21T01:47:18.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:17 smithi173 ceph-mon[25952]: Saving service rgw.foo spec with placement count:2 2024-06-21T01:47:18.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:17 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:18.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:17 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:18.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:17 smithi173 ceph-mon[25952]: Deploying daemon rgw.foo.smithi092.lzbkvh on smithi092 2024-06-21T01:47:19.077 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:18 smithi092 ceph-mon[25107]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-06-21T01:47:19.078 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:18 smithi092 ceph-mon[29810]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-06-21T01:47:19.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:18 smithi173 ceph-mon[25952]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-06-21T01:47:20.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:20 smithi173 ceph-mon[25952]: pgmap v79: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 112 B/s wr, 0 op/s 2024-06-21T01:47:20.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:20 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:20.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:20 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:20.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:20 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:20.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:20 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:20.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:20 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:20.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:20 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:20.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:20 smithi173 ceph-mon[25952]: Deploying daemon rgw.foo.smithi173.hvgnvt on smithi173 2024-06-21T01:47:20.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[29810]: pgmap v79: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 112 B/s wr, 0 op/s 2024-06-21T01:47:20.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:20.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:20.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:20.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:20.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:20.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:20.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[29810]: Deploying daemon rgw.foo.smithi173.hvgnvt on smithi173 2024-06-21T01:47:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[25107]: pgmap v79: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 112 B/s wr, 0 op/s 2024-06-21T01:47:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:20.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:20 smithi092 ceph-mon[25107]: Deploying daemon rgw.foo.smithi173.hvgnvt on smithi173 2024-06-21T01:47:22.668 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:47:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:47:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:47:22.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[29810]: pgmap v80: 161 pgs: 161 active+clean; 581 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-06-21T01:47:22.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:22.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[29810]: Saving service rgw.smpl spec with placement count:2 2024-06-21T01:47:22.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:22.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[29810]: Deploying daemon rgw.smpl.smithi092.bxxcjp on smithi092 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[25107]: pgmap v80: 161 pgs: 161 active+clean; 581 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[25107]: Saving service rgw.smpl spec with placement count:2 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:22.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:22.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:22.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:22 smithi092 ceph-mon[25107]: Deploying daemon rgw.smpl.smithi092.bxxcjp on smithi092 2024-06-21T01:47:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:22 smithi173 ceph-mon[25952]: pgmap v80: 161 pgs: 161 active+clean; 581 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-06-21T01:47:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:22 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:22 smithi173 ceph-mon[25952]: Saving service rgw.smpl spec with placement count:2 2024-06-21T01:47:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:22 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:22 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:22 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:22 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:22 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:22 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:22 smithi173 ceph-mon[25952]: Deploying daemon rgw.smpl.smithi092.bxxcjp on smithi092 2024-06-21T01:47:23.969 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:47:23] "GET /metrics HTTP/1.1" 200 206389 "" "Prometheus/2.33.4" 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:47:23.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:47:23.971 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:47:23.972 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:47:23.972 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-06-21T01:47:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-06-21T01:47:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-06-21T01:47:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-06-21T01:47:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-06-21T01:47:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:47:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:47:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:47:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:47:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:24.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: pgmap v81: 161 pgs: 161 active+clean; 581 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-06-21T01:47:24.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-06-21T01:47:24.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-06-21T01:47:24.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-06-21T01:47:24.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-06-21T01:47:24.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: osdmap e79: 8 total, 8 up, 8 in 2024-06-21T01:47:24.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:24.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:24.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:24.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:24.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:24.911 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:24 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: pgmap v81: 161 pgs: 161 active+clean; 581 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-06-21T01:47:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-06-21T01:47:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-06-21T01:47:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-06-21T01:47:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-06-21T01:47:25.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: osdmap e79: 8 total, 8 up, 8 in 2024-06-21T01:47:25.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:25.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:25.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:25.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:25.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:25.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:25.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: pgmap v81: 161 pgs: 161 active+clean; 581 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-06-21T01:47:25.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-06-21T01:47:25.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-06-21T01:47:25.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-06-21T01:47:25.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-06-21T01:47:25.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: osdmap e79: 8 total, 8 up, 8 in 2024-06-21T01:47:25.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:25.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:25.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:47:25.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-21T01:47:25.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:25.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:24 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:25 smithi092 ceph-mon[25107]: Deploying daemon rgw.smpl.smithi173.khwpnd on smithi173 2024-06-21T01:47:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:25 smithi092 ceph-mon[25107]: osdmap e80: 8 total, 8 up, 8 in 2024-06-21T01:47:25.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:25 smithi092 ceph-mon[29810]: Deploying daemon rgw.smpl.smithi173.khwpnd on smithi173 2024-06-21T01:47:25.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:25 smithi092 ceph-mon[29810]: osdmap e80: 8 total, 8 up, 8 in 2024-06-21T01:47:25.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:25 smithi173 ceph-mon[25952]: Deploying daemon rgw.smpl.smithi173.khwpnd on smithi173 2024-06-21T01:47:25.919 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:25 smithi173 ceph-mon[25952]: osdmap e80: 8 total, 8 up, 8 in 2024-06-21T01:47:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[25107]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 4.2 KiB/s wr, 359 op/s 2024-06-21T01:47:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:47:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:47:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-06-21T01:47:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[29810]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 4.2 KiB/s wr, 359 op/s 2024-06-21T01:47:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:47:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:47:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-06-21T01:47:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:26 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:26 smithi173 ceph-mon[25952]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 4.2 KiB/s wr, 359 op/s 2024-06-21T01:47:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:26 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:26 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:47:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:26 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:47:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:26 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-06-21T01:47:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:26 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:27.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:27 smithi173 ceph-mon[25952]: Checking pool "foo" exists for service iscsi.foo 2024-06-21T01:47:27.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:27 smithi173 ceph-mon[25952]: Deploying daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:47:28.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:27 smithi092 ceph-mon[25107]: Checking pool "foo" exists for service iscsi.foo 2024-06-21T01:47:28.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:27 smithi092 ceph-mon[25107]: Deploying daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:47:28.026 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:27 smithi092 ceph-mon[29810]: Checking pool "foo" exists for service iscsi.foo 2024-06-21T01:47:28.026 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:27 smithi092 ceph-mon[29810]: Deploying daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:47:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:28 smithi173 ceph-mon[25952]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 4.2 KiB/s wr, 359 op/s 2024-06-21T01:47:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:28 smithi173 ceph-mon[25952]: mgrmap e20: y(active, since 2m), standbys: x 2024-06-21T01:47:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:28 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:29.009 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:28 smithi092 ceph-mon[25107]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 4.2 KiB/s wr, 359 op/s 2024-06-21T01:47:29.009 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:28 smithi092 ceph-mon[25107]: mgrmap e20: y(active, since 2m), standbys: x 2024-06-21T01:47:29.009 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:28 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:29.009 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:28 smithi092 ceph-mon[29810]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 4.2 KiB/s wr, 359 op/s 2024-06-21T01:47:29.009 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:28 smithi092 ceph-mon[29810]: mgrmap e20: y(active, since 2m), standbys: x 2024-06-21T01:47:29.010 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:28 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:30 smithi173 ceph-mon[25952]: pgmap v86: 161 pgs: 2 peering, 159 active+clean; 584 KiB data, 79 MiB used, 715 GiB / 715 GiB avail; 340 KiB/s rd, 4.5 KiB/s wr, 575 op/s; 18 B/s, 0 objects/s recovering 2024-06-21T01:47:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:30 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:30 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:30 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:30 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2619227560' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:47:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:30 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2422138360' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4190712464"}]: dispatch 2024-06-21T01:47:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[25107]: pgmap v86: 161 pgs: 2 peering, 159 active+clean; 584 KiB data, 79 MiB used, 715 GiB / 715 GiB avail; 340 KiB/s rd, 4.5 KiB/s wr, 575 op/s; 18 B/s, 0 objects/s recovering 2024-06-21T01:47:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2619227560' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:47:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2422138360' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4190712464"}]: dispatch 2024-06-21T01:47:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[29810]: pgmap v86: 161 pgs: 2 peering, 159 active+clean; 584 KiB data, 79 MiB used, 715 GiB / 715 GiB avail; 340 KiB/s rd, 4.5 KiB/s wr, 575 op/s; 18 B/s, 0 objects/s recovering 2024-06-21T01:47:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2619227560' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:47:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:30 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2422138360' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4190712464"}]: dispatch 2024-06-21T01:47:31.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:31 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2422138360' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4190712464"}]': finished 2024-06-21T01:47:31.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:31 smithi173 ceph-mon[25952]: osdmap e81: 8 total, 8 up, 8 in 2024-06-21T01:47:31.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:31 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3972896832' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4190712464"}]: dispatch 2024-06-21T01:47:32.091 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:31 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2422138360' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4190712464"}]': finished 2024-06-21T01:47:32.091 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:31 smithi092 ceph-mon[25107]: osdmap e81: 8 total, 8 up, 8 in 2024-06-21T01:47:32.091 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:31 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3972896832' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4190712464"}]: dispatch 2024-06-21T01:47:32.091 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:31 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2422138360' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4190712464"}]': finished 2024-06-21T01:47:32.091 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:31 smithi092 ceph-mon[29810]: osdmap e81: 8 total, 8 up, 8 in 2024-06-21T01:47:32.091 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:31 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3972896832' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4190712464"}]: dispatch 2024-06-21T01:47:32.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:47:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:47:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:47:33.036 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:32 smithi173 ceph-mon[25952]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 468 KiB/s rd, 5.0 KiB/s wr, 776 op/s; 20 B/s, 0 objects/s recovering 2024-06-21T01:47:33.036 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:32 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3972896832' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4190712464"}]': finished 2024-06-21T01:47:33.036 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:32 smithi173 ceph-mon[25952]: osdmap e82: 8 total, 8 up, 8 in 2024-06-21T01:47:33.036 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:32 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3593780792' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4042387550"}]: dispatch 2024-06-21T01:47:33.036 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:32 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4042387550"}]: dispatch 2024-06-21T01:47:33.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:32 smithi092 ceph-mon[25107]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 468 KiB/s rd, 5.0 KiB/s wr, 776 op/s; 20 B/s, 0 objects/s recovering 2024-06-21T01:47:33.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:32 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3972896832' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4190712464"}]': finished 2024-06-21T01:47:33.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:32 smithi092 ceph-mon[25107]: osdmap e82: 8 total, 8 up, 8 in 2024-06-21T01:47:33.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:32 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3593780792' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4042387550"}]: dispatch 2024-06-21T01:47:33.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:32 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4042387550"}]: dispatch 2024-06-21T01:47:33.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:32 smithi092 ceph-mon[29810]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 468 KiB/s rd, 5.0 KiB/s wr, 776 op/s; 20 B/s, 0 objects/s recovering 2024-06-21T01:47:33.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:32 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3972896832' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4190712464"}]': finished 2024-06-21T01:47:33.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:32 smithi092 ceph-mon[29810]: osdmap e82: 8 total, 8 up, 8 in 2024-06-21T01:47:33.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:32 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3593780792' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4042387550"}]: dispatch 2024-06-21T01:47:33.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:32 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4042387550"}]: dispatch 2024-06-21T01:47:33.946 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4042387550"}]': finished 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[25107]: osdmap e83: 8 total, 8 up, 8 in 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2404890639' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/333757986"}]: dispatch 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/333757986"}]: dispatch 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4042387550"}]': finished 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[29810]: osdmap e83: 8 total, 8 up, 8 in 2024-06-21T01:47:33.947 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2404890639' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/333757986"}]: dispatch 2024-06-21T01:47:33.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/333757986"}]: dispatch 2024-06-21T01:47:33.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:33.948 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:47:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:47:33] "GET /metrics HTTP/1.1" 200 206389 "" "Prometheus/2.33.4" 2024-06-21T01:47:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:33 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4042387550"}]': finished 2024-06-21T01:47:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:33 smithi173 ceph-mon[25952]: osdmap e83: 8 total, 8 up, 8 in 2024-06-21T01:47:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:33 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:33 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2404890639' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/333757986"}]: dispatch 2024-06-21T01:47:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:33 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/333757986"}]: dispatch 2024-06-21T01:47:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:33 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:33 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:33 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:33 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:33 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:35.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[25107]: pgmap v91: 161 pgs: 161 active+clean; 584 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 293 KiB/s rd, 341 B/s wr, 444 op/s; 24 B/s, 0 objects/s recovering 2024-06-21T01:47:35.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[25107]: Checking dashboard <-> RGW credentials 2024-06-21T01:47:35.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[25107]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-21T01:47:35.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[25107]: Reconfiguring daemon alertmanager.a on smithi092 2024-06-21T01:47:35.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/333757986"}]': finished 2024-06-21T01:47:35.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[25107]: osdmap e84: 8 total, 8 up, 8 in 2024-06-21T01:47:35.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/991122965' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/244652261"}]: dispatch 2024-06-21T01:47:35.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/244652261"}]: dispatch 2024-06-21T01:47:35.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[29810]: pgmap v91: 161 pgs: 161 active+clean; 584 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 293 KiB/s rd, 341 B/s wr, 444 op/s; 24 B/s, 0 objects/s recovering 2024-06-21T01:47:35.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[29810]: Checking dashboard <-> RGW credentials 2024-06-21T01:47:35.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[29810]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-21T01:47:35.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[29810]: Reconfiguring daemon alertmanager.a on smithi092 2024-06-21T01:47:35.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/333757986"}]': finished 2024-06-21T01:47:35.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[29810]: osdmap e84: 8 total, 8 up, 8 in 2024-06-21T01:47:35.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/991122965' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/244652261"}]: dispatch 2024-06-21T01:47:35.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:34 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/244652261"}]: dispatch 2024-06-21T01:47:35.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:34 smithi173 ceph-mon[25952]: pgmap v91: 161 pgs: 161 active+clean; 584 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 293 KiB/s rd, 341 B/s wr, 444 op/s; 24 B/s, 0 objects/s recovering 2024-06-21T01:47:35.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:34 smithi173 ceph-mon[25952]: Checking dashboard <-> RGW credentials 2024-06-21T01:47:35.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:34 smithi173 ceph-mon[25952]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-21T01:47:35.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:34 smithi173 ceph-mon[25952]: Reconfiguring daemon alertmanager.a on smithi092 2024-06-21T01:47:35.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:34 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/333757986"}]': finished 2024-06-21T01:47:35.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:34 smithi173 ceph-mon[25952]: osdmap e84: 8 total, 8 up, 8 in 2024-06-21T01:47:35.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:34 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/991122965' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/244652261"}]: dispatch 2024-06-21T01:47:35.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:34 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/244652261"}]: dispatch 2024-06-21T01:47:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:35 smithi092 ceph-mon[25107]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 751 B/s wr, 116 op/s 2024-06-21T01:47:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:35 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/244652261"}]': finished 2024-06-21T01:47:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:35 smithi092 ceph-mon[25107]: osdmap e85: 8 total, 8 up, 8 in 2024-06-21T01:47:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:35 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1751463756' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1798744344"}]: dispatch 2024-06-21T01:47:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:35 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1798744344"}]: dispatch 2024-06-21T01:47:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:35 smithi092 ceph-mon[29810]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 751 B/s wr, 116 op/s 2024-06-21T01:47:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:35 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/244652261"}]': finished 2024-06-21T01:47:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:35 smithi092 ceph-mon[29810]: osdmap e85: 8 total, 8 up, 8 in 2024-06-21T01:47:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:35 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1751463756' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1798744344"}]: dispatch 2024-06-21T01:47:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:35 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1798744344"}]: dispatch 2024-06-21T01:47:36.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:35 smithi173 ceph-mon[25952]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 751 B/s wr, 116 op/s 2024-06-21T01:47:36.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:35 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/244652261"}]': finished 2024-06-21T01:47:36.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:35 smithi173 ceph-mon[25952]: osdmap e85: 8 total, 8 up, 8 in 2024-06-21T01:47:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:35 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1751463756' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1798744344"}]: dispatch 2024-06-21T01:47:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:35 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1798744344"}]: dispatch 2024-06-21T01:47:36.927 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:36 smithi092 systemd[1]: Stopping Ceph alertmanager.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:47:36.927 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:36 smithi092 bash[53250]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager.a" found: no such container 2024-06-21T01:47:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:36 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1798744344"}]': finished 2024-06-21T01:47:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:36 smithi173 ceph-mon[25952]: osdmap e86: 8 total, 8 up, 8 in 2024-06-21T01:47:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:36 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1153223889' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4223667356"}]: dispatch 2024-06-21T01:47:37.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:36 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1798744344"}]': finished 2024-06-21T01:47:37.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:36 smithi092 ceph-mon[29810]: osdmap e86: 8 total, 8 up, 8 in 2024-06-21T01:47:37.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:36 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1153223889' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4223667356"}]: dispatch 2024-06-21T01:47:37.270 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:36 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1798744344"}]': finished 2024-06-21T01:47:37.270 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:36 smithi092 ceph-mon[25107]: osdmap e86: 8 total, 8 up, 8 in 2024-06-21T01:47:37.271 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:36 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1153223889' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4223667356"}]: dispatch 2024-06-21T01:47:37.271 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:36 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[48084]: level=info ts=2024-06-21T01:47:36.930Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-06-21T01:47:37.271 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:36 smithi092 podman[53261]: 2024-06-21 01:47:36.971675714 +0000 UTC m=+0.155928550 container died e7db2ea37246f7729883f8bf80e746fdbbd835bbead7c95e57ce86ebe3a52519 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:37.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:37 smithi092 podman[53261]: 2024-06-21 01:47:37.45594984 +0000 UTC m=+0.640202674 container cleanup e7db2ea37246f7729883f8bf80e746fdbbd835bbead7c95e57ce86ebe3a52519 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:37.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:37 smithi092 bash[53261]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a 2024-06-21T01:47:37.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:37 smithi092 bash[53318]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager.a" found: no such container 2024-06-21T01:47:37.950 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:37 smithi092 podman[53275]: 2024-06-21 01:47:37.690375909 +0000 UTC m=+0.713381319 container remove e7db2ea37246f7729883f8bf80e746fdbbd835bbead7c95e57ce86ebe3a52519 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:37.950 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:37 smithi092 podman[53275]: 2024-06-21 01:47:37.766111639 +0000 UTC m=+0.789117050 volume remove 7da797bb50a5d177b23bf00b1ce3c6d4d77fe18c03fe923ccc5bfd5edafd25c7 2024-06-21T01:47:37.950 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:37 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-06-21T01:47:37.950 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:37 smithi092 systemd[1]: Stopped Ceph alertmanager.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:47:37.950 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:37 smithi092 systemd[1]: Starting Ceph alertmanager.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:47:38.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:37 smithi173 ceph-mon[25952]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 753 B/s wr, 117 op/s 2024-06-21T01:47:38.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:37 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1153223889' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4223667356"}]': finished 2024-06-21T01:47:38.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:37 smithi173 ceph-mon[25952]: osdmap e87: 8 total, 8 up, 8 in 2024-06-21T01:47:38.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:37 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/182063774' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2316531354"}]: dispatch 2024-06-21T01:47:38.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:37 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2316531354"}]: dispatch 2024-06-21T01:47:38.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:37 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:38.269 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[29810]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 753 B/s wr, 117 op/s 2024-06-21T01:47:38.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1153223889' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4223667356"}]': finished 2024-06-21T01:47:38.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[29810]: osdmap e87: 8 total, 8 up, 8 in 2024-06-21T01:47:38.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/182063774' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2316531354"}]: dispatch 2024-06-21T01:47:38.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2316531354"}]: dispatch 2024-06-21T01:47:38.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:38.270 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[25107]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 753 B/s wr, 117 op/s 2024-06-21T01:47:38.270 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1153223889' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4223667356"}]': finished 2024-06-21T01:47:38.270 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[25107]: osdmap e87: 8 total, 8 up, 8 in 2024-06-21T01:47:38.271 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/182063774' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2316531354"}]: dispatch 2024-06-21T01:47:38.271 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2316531354"}]: dispatch 2024-06-21T01:47:38.271 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:37 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:38.271 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:37 smithi092 podman[53383]: 2024-06-21 01:47:37.895483233 +0000 UTC m=+0.018162705 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-06-21T01:47:38.271 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 podman[53383]: 2024-06-21 01:47:38.000420295 +0000 UTC m=+0.123099757 volume create 05f6843a2454d4d550abc508badf9820914e81ed64d2abd6421a51f2b073a3d7 2024-06-21T01:47:38.271 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 podman[53383]: 2024-06-21 01:47:38.109114462 +0000 UTC m=+0.231793913 container create 510d745d0b554cb82c22197bce7f5486ec2e76a49ac1234c3dadbe088a8093a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:38.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 podman[53383]: 2024-06-21 01:47:38.362056591 +0000 UTC m=+0.484736049 container init 510d745d0b554cb82c22197bce7f5486ec2e76a49ac1234c3dadbe088a8093a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:38.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 podman[53383]: 2024-06-21 01:47:38.364843153 +0000 UTC m=+0.487522606 container start 510d745d0b554cb82c22197bce7f5486ec2e76a49ac1234c3dadbe088a8093a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:38.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=info ts=2024-06-21T01:47:38.373Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-06-21T01:47:38.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=info ts=2024-06-21T01:47:38.373Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-06-21T01:47:38.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=info ts=2024-06-21T01:47:38.374Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.92 port=9094 2024-06-21T01:47:38.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=info ts=2024-06-21T01:47:38.376Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-06-21T01:47:38.852 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=info ts=2024-06-21T01:47:38.401Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-21T01:47:38.852 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=info ts=2024-06-21T01:47:38.401Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-21T01:47:38.852 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=info ts=2024-06-21T01:47:38.403Z caller=main.go:518 msg=Listening address=:9093 2024-06-21T01:47:38.852 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=info ts=2024-06-21T01:47:38.404Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-06-21T01:47:38.852 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 bash[53383]: 510d745d0b554cb82c22197bce7f5486ec2e76a49ac1234c3dadbe088a8093a0 2024-06-21T01:47:38.852 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:38 smithi092 systemd[1]: Started Ceph alertmanager.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:47:39.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:38 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2316531354"}]': finished 2024-06-21T01:47:39.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:38 smithi173 ceph-mon[25952]: osdmap e88: 8 total, 8 up, 8 in 2024-06-21T01:47:39.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:38 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/952272522' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2316531354"}]: dispatch 2024-06-21T01:47:39.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:38 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2316531354"}]: dispatch 2024-06-21T01:47:39.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:38 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:39.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:38 smithi173 ceph-mon[25952]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-21T01:47:39.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:38 smithi173 ceph-mon[25952]: Reconfiguring daemon prometheus.a on smithi173 2024-06-21T01:47:39.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2316531354"}]': finished 2024-06-21T01:47:39.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[25107]: osdmap e88: 8 total, 8 up, 8 in 2024-06-21T01:47:39.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/952272522' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2316531354"}]: dispatch 2024-06-21T01:47:39.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2316531354"}]: dispatch 2024-06-21T01:47:39.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:39.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[25107]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-21T01:47:39.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[25107]: Reconfiguring daemon prometheus.a on smithi173 2024-06-21T01:47:39.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2316531354"}]': finished 2024-06-21T01:47:39.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[29810]: osdmap e88: 8 total, 8 up, 8 in 2024-06-21T01:47:39.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/952272522' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2316531354"}]: dispatch 2024-06-21T01:47:39.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2316531354"}]: dispatch 2024-06-21T01:47:39.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:39.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[29810]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-21T01:47:39.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:38 smithi092 ceph-mon[29810]: Reconfiguring daemon prometheus.a on smithi173 2024-06-21T01:47:40.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:39 smithi092 ceph-mon[25107]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 31 op/s 2024-06-21T01:47:40.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:39 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2316531354"}]': finished 2024-06-21T01:47:40.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:39 smithi092 ceph-mon[25107]: osdmap e89: 8 total, 8 up, 8 in 2024-06-21T01:47:40.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:39 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/43636773' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4215008918"}]: dispatch 2024-06-21T01:47:40.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:39 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4215008918"}]: dispatch 2024-06-21T01:47:40.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:39 smithi092 ceph-mon[29810]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 31 op/s 2024-06-21T01:47:40.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:39 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2316531354"}]': finished 2024-06-21T01:47:40.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:39 smithi092 ceph-mon[29810]: osdmap e89: 8 total, 8 up, 8 in 2024-06-21T01:47:40.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:39 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/43636773' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4215008918"}]: dispatch 2024-06-21T01:47:40.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:39 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4215008918"}]: dispatch 2024-06-21T01:47:40.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:39 smithi173 ceph-mon[25952]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 31 op/s 2024-06-21T01:47:40.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:39 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2316531354"}]': finished 2024-06-21T01:47:40.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:39 smithi173 ceph-mon[25952]: osdmap e89: 8 total, 8 up, 8 in 2024-06-21T01:47:40.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:39 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/43636773' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4215008918"}]: dispatch 2024-06-21T01:47:40.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:39 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4215008918"}]: dispatch 2024-06-21T01:47:40.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=info ts=2024-06-21T01:47:40.377Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000671868s 2024-06-21T01:47:41.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:40 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:47:41.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:40 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4215008918"}]': finished 2024-06-21T01:47:41.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:40 smithi173 ceph-mon[25952]: osdmap e90: 8 total, 8 up, 8 in 2024-06-21T01:47:41.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:40 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/696899025' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1379486465"}]: dispatch 2024-06-21T01:47:41.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:40 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1379486465"}]: dispatch 2024-06-21T01:47:41.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:40 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:47:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:40 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4215008918"}]': finished 2024-06-21T01:47:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:40 smithi092 ceph-mon[29810]: osdmap e90: 8 total, 8 up, 8 in 2024-06-21T01:47:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:40 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/696899025' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1379486465"}]: dispatch 2024-06-21T01:47:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:40 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1379486465"}]: dispatch 2024-06-21T01:47:41.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:40 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:47:41.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:40 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/4215008918"}]': finished 2024-06-21T01:47:41.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:40 smithi092 ceph-mon[25107]: osdmap e90: 8 total, 8 up, 8 in 2024-06-21T01:47:41.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:40 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/696899025' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1379486465"}]: dispatch 2024-06-21T01:47:41.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:40 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1379486465"}]: dispatch 2024-06-21T01:47:41.561 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 systemd[1]: Stopping Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:47:41.562 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 bash[44879]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus.a" found: no such container 2024-06-21T01:47:41.911 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-21T01:47:41.911 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-06-21T01:47:41.911 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[40991]: ts=2024-06-21T01:47:41.565Z caller=main.go:1066 level=info msg="See you next time!" 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 podman[44892]: 2024-06-21 01:47:41.575290708 +0000 UTC m=+0.126836404 container died 13533f3524e0c189a721bffbe01b9c1092f2f6756a484b9ada60b10a52dc73d1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 podman[44892]: 2024-06-21 01:47:41.914821822 +0000 UTC m=+0.466367519 container cleanup 13533f3524e0c189a721bffbe01b9c1092f2f6756a484b9ada60b10a52dc73d1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:41.912 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:41 smithi173 bash[44892]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a 2024-06-21T01:47:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-mon[25952]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 37 op/s 2024-06-21T01:47:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1379486465"}]': finished 2024-06-21T01:47:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-mon[25952]: osdmap e91: 8 total, 8 up, 8 in 2024-06-21T01:47:42.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:41 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/926436247' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3774566714"}]: dispatch 2024-06-21T01:47:42.229 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 podman[44904]: 2024-06-21 01:47:42.10787596 +0000 UTC m=+0.533605815 container remove 13533f3524e0c189a721bffbe01b9c1092f2f6756a484b9ada60b10a52dc73d1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:42.229 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 bash[44918]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus.a" found: no such container 2024-06-21T01:47:42.229 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-06-21T01:47:42.229 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 systemd[1]: Stopped Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:47:42.229 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 systemd[1]: Starting Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:47:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:41 smithi092 ceph-mon[25107]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 37 op/s 2024-06-21T01:47:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:41 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1379486465"}]': finished 2024-06-21T01:47:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:41 smithi092 ceph-mon[25107]: osdmap e91: 8 total, 8 up, 8 in 2024-06-21T01:47:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:41 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/926436247' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3774566714"}]: dispatch 2024-06-21T01:47:42.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:41 smithi092 ceph-mon[29810]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 37 op/s 2024-06-21T01:47:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:41 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1379486465"}]': finished 2024-06-21T01:47:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:41 smithi092 ceph-mon[29810]: osdmap e91: 8 total, 8 up, 8 in 2024-06-21T01:47:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:41 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/926436247' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3774566714"}]: dispatch 2024-06-21T01:47:42.641 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 podman[44985]: 2024-06-21 01:47:42.350843585 +0000 UTC m=+0.110705926 container create c36e844319fcfb19a2bb3674c9b8fa8f181b5bb8231d37105043b53b50d2467e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:42.641 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 podman[44985]: 2024-06-21 01:47:42.258259969 +0000 UTC m=+0.018122317 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 podman[44985]: 2024-06-21 01:47:42.553359267 +0000 UTC m=+0.313221621 container init c36e844319fcfb19a2bb3674c9b8fa8f181b5bb8231d37105043b53b50d2467e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 podman[44985]: 2024-06-21 01:47:42.556363855 +0000 UTC m=+0.316226197 container start c36e844319fcfb19a2bb3674c9b8fa8f181b5bb8231d37105043b53b50d2467e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.580Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.580Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.580Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.580Z caller=main.go:518 level=info host_details="(Linux 5.14.0-460.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024 x86_64 smithi173 (none))" 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.580Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.580Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.582Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.582Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.583Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-06-21T01:47:42.642 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.586Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-06-21T01:47:42.643 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.586Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.869µs 2024-06-21T01:47:42.643 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.586Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-06-21T01:47:42.643 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 bash[44985]: c36e844319fcfb19a2bb3674c9b8fa8f181b5bb8231d37105043b53b50d2467e 2024-06-21T01:47:42.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 systemd[1]: Started Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:47:42.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.819Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-06-21T01:47:42.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.819Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-06-21T01:47:42.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.819Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=35.197µs wal_replay_duration=233.288086ms total_replay_duration=233.337278ms 2024-06-21T01:47:42.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.821Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-06-21T01:47:42.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.821Z caller=main.go:947 level=info msg="TSDB started" 2024-06-21T01:47:42.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.821Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-06-21T01:47:42.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.835Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=13.79406ms db_storage=1.15µs remote_storage=1.289µs web_handler=309ns query_engine=592ns scrape=233.288µs scrape_sd=22.593µs notify=19.68µs notify_sd=6.146µs rules=13.252631ms 2024-06-21T01:47:42.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:47:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:47:42.835Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-06-21T01:47:43.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/926436247' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3774566714"}]': finished 2024-06-21T01:47:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: osdmap e92: 8 total, 8 up, 8 in 2024-06-21T01:47:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1318812369' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1521357991"}]: dispatch 2024-06-21T01:47:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:47:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:47:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-06-21T01:47:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-06-21T01:47:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-21T01:47:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-21T01:47:43.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: Adding iSCSI gateway http://:@172.21.15.92:5000 to Dashboard 2024-06-21T01:47:43.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-21T01:47:43.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-21T01:47:43.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-06-21T01:47:43.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-06-21T01:47:43.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:47:43.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:47:43.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.173:9095"}]: dispatch 2024-06-21T01:47:43.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.173:9095"}]: dispatch 2024-06-21T01:47:43.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:47:43.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:47:43.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.173:3000"}]: dispatch 2024-06-21T01:47:43.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.173:3000"}]: dispatch 2024-06-21T01:47:43.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:43.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:43.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/926436247' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3774566714"}]': finished 2024-06-21T01:47:43.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: osdmap e92: 8 total, 8 up, 8 in 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1318812369' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1521357991"}]: dispatch 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: Adding iSCSI gateway http://:@172.21.15.92:5000 to Dashboard 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-06-21T01:47:43.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.173:9095"}]: dispatch 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.173:9095"}]: dispatch 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.173:3000"}]: dispatch 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.173:3000"}]: dispatch 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:43.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:43 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/926436247' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3774566714"}]': finished 2024-06-21T01:47:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: osdmap e92: 8 total, 8 up, 8 in 2024-06-21T01:47:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1318812369' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1521357991"}]: dispatch 2024-06-21T01:47:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:47:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:47:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-06-21T01:47:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-06-21T01:47:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: Adding iSCSI gateway http://:@172.21.15.92:5000 to Dashboard 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.173:9095"}]: dispatch 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.173:9095"}]: dispatch 2024-06-21T01:47:43.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:47:43.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:47:43.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.173:3000"}]: dispatch 2024-06-21T01:47:43.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.173:3000"}]: dispatch 2024-06-21T01:47:43.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:43.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:43.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:43 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:44.076 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:44 smithi173 ceph-mon[25952]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-06-21T01:47:44.076 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:44 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1318812369' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1521357991"}]': finished 2024-06-21T01:47:44.076 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:44 smithi173 ceph-mon[25952]: osdmap e93: 8 total, 8 up, 8 in 2024-06-21T01:47:44.077 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:44 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/4121127224' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2570230159"}]: dispatch 2024-06-21T01:47:44.077 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:44 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2570230159"}]: dispatch 2024-06-21T01:47:44.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:44 smithi092 ceph-mon[29810]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-06-21T01:47:44.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:44 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1318812369' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1521357991"}]': finished 2024-06-21T01:47:44.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:44 smithi092 ceph-mon[29810]: osdmap e93: 8 total, 8 up, 8 in 2024-06-21T01:47:44.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:44 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/4121127224' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2570230159"}]: dispatch 2024-06-21T01:47:44.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:44 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2570230159"}]: dispatch 2024-06-21T01:47:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:44 smithi092 ceph-mon[25107]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-06-21T01:47:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:44 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1318812369' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1521357991"}]': finished 2024-06-21T01:47:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:44 smithi092 ceph-mon[25107]: osdmap e93: 8 total, 8 up, 8 in 2024-06-21T01:47:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:44 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/4121127224' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2570230159"}]: dispatch 2024-06-21T01:47:44.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:44 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2570230159"}]: dispatch 2024-06-21T01:47:45.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:45 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2570230159"}]': finished 2024-06-21T01:47:45.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:45 smithi173 ceph-mon[25952]: osdmap e94: 8 total, 8 up, 8 in 2024-06-21T01:47:45.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:45 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3230241683' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/648013728"}]: dispatch 2024-06-21T01:47:45.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:45 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/648013728"}]: dispatch 2024-06-21T01:47:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:45 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2570230159"}]': finished 2024-06-21T01:47:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:45 smithi092 ceph-mon[25107]: osdmap e94: 8 total, 8 up, 8 in 2024-06-21T01:47:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:45 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3230241683' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/648013728"}]: dispatch 2024-06-21T01:47:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:45 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/648013728"}]: dispatch 2024-06-21T01:47:45.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:45 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2570230159"}]': finished 2024-06-21T01:47:45.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:45 smithi092 ceph-mon[29810]: osdmap e94: 8 total, 8 up, 8 in 2024-06-21T01:47:45.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:45 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3230241683' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/648013728"}]: dispatch 2024-06-21T01:47:45.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:45 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/648013728"}]: dispatch 2024-06-21T01:47:46.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:46 smithi173 ceph-mon[25952]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:47:46.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:46 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/648013728"}]': finished 2024-06-21T01:47:46.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:46 smithi173 ceph-mon[25952]: osdmap e95: 8 total, 8 up, 8 in 2024-06-21T01:47:46.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:46 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/559663940' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/648013728"}]: dispatch 2024-06-21T01:47:46.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:46 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/648013728"}]: dispatch 2024-06-21T01:47:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:46 smithi092 ceph-mon[25107]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:47:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:46 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/648013728"}]': finished 2024-06-21T01:47:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:46 smithi092 ceph-mon[25107]: osdmap e95: 8 total, 8 up, 8 in 2024-06-21T01:47:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:46 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/559663940' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/648013728"}]: dispatch 2024-06-21T01:47:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:46 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/648013728"}]: dispatch 2024-06-21T01:47:46.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:46 smithi092 ceph-mon[29810]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:47:46.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:46 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/648013728"}]': finished 2024-06-21T01:47:46.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:46 smithi092 ceph-mon[29810]: osdmap e95: 8 total, 8 up, 8 in 2024-06-21T01:47:46.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:46 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/559663940' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/648013728"}]: dispatch 2024-06-21T01:47:46.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:46 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/648013728"}]: dispatch 2024-06-21T01:47:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/648013728"}]': finished 2024-06-21T01:47:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[25107]: osdmap e96: 8 total, 8 up, 8 in 2024-06-21T01:47:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[25107]: Checking dashboard <-> RGW credentials 2024-06-21T01:47:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:47.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:47.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/648013728"}]': finished 2024-06-21T01:47:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[29810]: osdmap e96: 8 total, 8 up, 8 in 2024-06-21T01:47:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[29810]: Checking dashboard <-> RGW credentials 2024-06-21T01:47:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:47 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:47.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:47 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/648013728"}]': finished 2024-06-21T01:47:47.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:47 smithi173 ceph-mon[25952]: osdmap e96: 8 total, 8 up, 8 in 2024-06-21T01:47:47.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:47 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:47.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:47 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:47.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:47 smithi173 ceph-mon[25952]: Checking dashboard <-> RGW credentials 2024-06-21T01:47:47.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:47 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:47.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:47 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:47:47.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:47 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:47:48.464 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:47:48 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=info ts=2024-06-21T01:47:48.379Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002667079s 2024-06-21T01:47:48.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:48 smithi092 ceph-mon[25107]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:47:48.723 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:48 smithi092 ceph-mon[29810]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:47:48.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:48 smithi173 ceph-mon[25952]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:47:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:50 smithi173 ceph-mon[25952]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 18 op/s 2024-06-21T01:47:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:50 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:50 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:50 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:51.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:50 smithi092 ceph-mon[29810]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 18 op/s 2024-06-21T01:47:51.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:50 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:51.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:50 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:51.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:50 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:51.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:50 smithi092 ceph-mon[25107]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 18 op/s 2024-06-21T01:47:51.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:50 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:51.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:50 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:51.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:50 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:47:51.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:51 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:47:52.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:51 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:47:52.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:51 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:47:52.683 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:47:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:47:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:47:52.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:52 smithi173 ceph-mon[25952]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 5.4 KiB/s rd, 5 op/s 2024-06-21T01:47:53.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:52 smithi092 ceph-mon[29810]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 5.4 KiB/s rd, 5 op/s 2024-06-21T01:47:53.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:52 smithi092 ceph-mon[25107]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 5.4 KiB/s rd, 5 op/s 2024-06-21T01:47:54.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:47:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:47:53] "GET /metrics HTTP/1.1" 200 215058 "" "Prometheus/2.33.4" 2024-06-21T01:47:54.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:54 smithi173 ceph-mon[25952]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 4 op/s 2024-06-21T01:47:55.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:54 smithi092 ceph-mon[25107]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 4 op/s 2024-06-21T01:47:55.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:54 smithi092 ceph-mon[29810]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 4 op/s 2024-06-21T01:47:56.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:56 smithi173 ceph-mon[25952]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 4 op/s 2024-06-21T01:47:57.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:56 smithi092 ceph-mon[25107]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 4 op/s 2024-06-21T01:47:57.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:56 smithi092 ceph-mon[29810]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 4 op/s 2024-06-21T01:47:58.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:47:58 smithi173 ceph-mon[25952]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 3 op/s 2024-06-21T01:47:59.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:47:58 smithi092 ceph-mon[25107]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 3 op/s 2024-06-21T01:47:59.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:47:58 smithi092 ceph-mon[29810]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 3 op/s 2024-06-21T01:48:00.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:00 smithi173 ceph-mon[25952]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 3 op/s 2024-06-21T01:48:01.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:00 smithi092 ceph-mon[25107]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 3 op/s 2024-06-21T01:48:01.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:00 smithi092 ceph-mon[29810]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 3 op/s 2024-06-21T01:48:01.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:01 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:02.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:01 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:02.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:01 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:02.704 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:48:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:48:02.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:02 smithi173 ceph-mon[25952]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:03.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:02 smithi092 ceph-mon[25107]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:03.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:02 smithi092 ceph-mon[29810]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:04.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:48:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:03] "GET /metrics HTTP/1.1" 200 215058 "" "Prometheus/2.33.4" 2024-06-21T01:48:04.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:04 smithi173 ceph-mon[25952]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:05.002 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:04 smithi092 ceph-mon[25107]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:05.002 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:04 smithi092 ceph-mon[29810]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:06.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:06 smithi173 ceph-mon[25952]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:07.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:06 smithi092 ceph-mon[25107]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:07.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:06 smithi092 ceph-mon[29810]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:08.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:08 smithi173 ceph-mon[25952]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:09.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:08 smithi092 ceph-mon[25107]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:09.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:08 smithi092 ceph-mon[29810]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:10.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:10 smithi173 ceph-mon[25952]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:11.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:10 smithi092 ceph-mon[25107]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:11.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:10 smithi092 ceph-mon[29810]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:11.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:11 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:12.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:11 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:12.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:11 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:12.721 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:48:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:48:12.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:12 smithi173 ceph-mon[25952]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:13.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:12 smithi092 ceph-mon[25107]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:13.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:12 smithi092 ceph-mon[29810]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:14.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:48:13 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:13] "GET /metrics HTTP/1.1" 200 214984 "" "Prometheus/2.33.4" 2024-06-21T01:48:14.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:14 smithi173 ceph-mon[25952]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:15.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:14 smithi092 ceph-mon[25107]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:15.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:14 smithi092 ceph-mon[29810]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:17.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:16 smithi092 ceph-mon[25107]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:17.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:16 smithi092 ceph-mon[29810]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:17.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:16 smithi173 ceph-mon[25952]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:19.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:18 smithi092 ceph-mon[25107]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:19.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:18 smithi092 ceph-mon[29810]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:19.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:18 smithi173 ceph-mon[25952]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:21.030 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:20 smithi173 ceph-mon[25952]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:21.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:20 smithi092 ceph-mon[25107]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:21.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:20 smithi092 ceph-mon[29810]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:22.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:21 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:22.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:21 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:22.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:21 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:22.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:48:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:48:23.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:22 smithi092 ceph-mon[25107]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:23.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:22 smithi092 ceph-mon[29810]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:23.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:22 smithi173 ceph-mon[25952]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:24.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:48:24.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:48:24.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:48:24.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:48:24.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:48:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:23] "GET /metrics HTTP/1.1" 200 214935 "" "Prometheus/2.33.4" 2024-06-21T01:48:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:48:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:48:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:48:24.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:48:24.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:48:24.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:48:24.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:48:24.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:48:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:24 smithi092 ceph-mon[25107]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:25.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:24 smithi092 ceph-mon[29810]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:25.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:24 smithi173 ceph-mon[25952]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:27.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:26 smithi092 ceph-mon[25107]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:27.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:26 smithi092 ceph-mon[29810]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:27.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:26 smithi173 ceph-mon[25952]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:29.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:28 smithi092 ceph-mon[25107]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:29.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:28 smithi092 ceph-mon[29810]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:29.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:28 smithi173 ceph-mon[25952]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:30 smithi092 ceph-mon[25107]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:31.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:30 smithi092 ceph-mon[29810]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:31.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:30 smithi173 ceph-mon[25952]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:32.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:31 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:32.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:31 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:32.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:31 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:32.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:48:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:48:33.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:32 smithi092 ceph-mon[25107]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:33.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:32 smithi092 ceph-mon[29810]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:33.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:32 smithi173 ceph-mon[25952]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:34.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:48:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:33] "GET /metrics HTTP/1.1" 200 214935 "" "Prometheus/2.33.4" 2024-06-21T01:48:35.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:34 smithi092 ceph-mon[29810]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:35.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:34 smithi092 ceph-mon[25107]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:35.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:34 smithi173 ceph-mon[25952]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:37.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:36 smithi092 ceph-mon[25107]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:37.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:36 smithi092 ceph-mon[29810]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:36 smithi173 ceph-mon[25952]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:39.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:38 smithi092 ceph-mon[25107]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:39.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:38 smithi092 ceph-mon[29810]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:39.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:38 smithi173 ceph-mon[25952]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:41.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:40 smithi092 ceph-mon[25107]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:41.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:40 smithi092 ceph-mon[29810]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:41.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:40 smithi173 ceph-mon[25952]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:42.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:41 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:42.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:41 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:41 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:42.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:48:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:48:43.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:42 smithi092 ceph-mon[25107]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:43.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:42 smithi092 ceph-mon[29810]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:43.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:42 smithi173 ceph-mon[25952]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:44.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:48:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:43] "GET /metrics HTTP/1.1" 200 214937 "" "Prometheus/2.33.4" 2024-06-21T01:48:45.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:44 smithi092 ceph-mon[25107]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:45.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:44 smithi092 ceph-mon[29810]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:45.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:44 smithi173 ceph-mon[25952]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:47.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:46 smithi092 ceph-mon[25107]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:47.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:46 smithi092 ceph-mon[29810]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:47.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:46 smithi173 ceph-mon[25952]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:49.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:48 smithi092 ceph-mon[25107]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:49.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:48 smithi092 ceph-mon[29810]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:49.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:48 smithi173 ceph-mon[25952]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:50.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:49 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:48:50.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:49 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:48:50.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:49 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:48:50.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:49 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:48:50.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:49 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:48:50.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:49 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:48:51.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:50 smithi092 ceph-mon[25107]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:51.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:50 smithi092 ceph-mon[29810]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:51.209 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:50 smithi173 ceph-mon[25952]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:52.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:51 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:52.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:51 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:52.122 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:51 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:48:52.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:48:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:48:53.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:52 smithi092 ceph-mon[25107]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:53.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:52 smithi092 ceph-mon[29810]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:53.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:52 smithi173 ceph-mon[25952]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:54.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:48:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:48:53] "GET /metrics HTTP/1.1" 200 214933 "" "Prometheus/2.33.4" 2024-06-21T01:48:54.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:54 smithi173 ceph-mon[25952]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:54.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:54 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:48:54.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:54 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:48:54.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:54 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:48:54.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:54 smithi092 ceph-mon[25107]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:54.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:54 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:48:54.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:54 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:48:54.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:54 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:48:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:54 smithi092 ceph-mon[29810]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:54 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:48:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:54 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:48:54.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:54 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:48:56.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:56 smithi173 ceph-mon[25952]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:56.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:56 smithi092 ceph-mon[25107]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:56.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:56 smithi092 ceph-mon[29810]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:48:58.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:48:58 smithi173 ceph-mon[25952]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:58.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:48:58 smithi092 ceph-mon[25107]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:48:58.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:48:58 smithi092 ceph-mon[29810]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:00.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:00 smithi173 ceph-mon[25952]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:00 smithi092 ceph-mon[25107]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:00.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:00 smithi092 ceph-mon[29810]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:01.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:01 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:01.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:01 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:01.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:01 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:02.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:02 smithi173 ceph-mon[25952]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:02.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:49:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:49:02.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:02 smithi092 ceph-mon[25107]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:02.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:02 smithi092 ceph-mon[29810]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:04.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:49:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:03] "GET /metrics HTTP/1.1" 200 214933 "" "Prometheus/2.33.4" 2024-06-21T01:49:04.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:04 smithi173 ceph-mon[25952]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:04.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:04 smithi092 ceph-mon[25107]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:04.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:04 smithi092 ceph-mon[29810]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:06.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:06 smithi173 ceph-mon[25952]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:06.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:06 smithi092 ceph-mon[25107]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:06.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:06 smithi092 ceph-mon[29810]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:08.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:08 smithi173 ceph-mon[25952]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:08.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:08 smithi092 ceph-mon[25107]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:08.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:08 smithi092 ceph-mon[29810]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:10.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:10 smithi092 ceph-mon[29810]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:10.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:10 smithi092 ceph-mon[25107]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:10.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:10 smithi173 ceph-mon[25952]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:11.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:11 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:11.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:11 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:11.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:11 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:12.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:12 smithi173 ceph-mon[25952]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:12.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:49:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:49:12.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:12 smithi092 ceph-mon[25107]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:12.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:12 smithi092 ceph-mon[29810]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:14.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:49:13 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:13] "GET /metrics HTTP/1.1" 200 214946 "" "Prometheus/2.33.4" 2024-06-21T01:49:14.784 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:14 smithi092 ceph-mon[25107]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:14.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:14 smithi092 ceph-mon[29810]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:14.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:14 smithi173 ceph-mon[25952]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:15.222 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-06-21T01:49:16.581 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:16 smithi092 ceph-mon[25107]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:16.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:16 smithi092 ceph-mon[29810]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:16.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:16 smithi173 ceph-mon[25952]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:17.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-06-21T01:49:18.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:18 smithi092 ceph-mon[25107]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:18.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:18 smithi092 ceph-mon[29810]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:18.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:18 smithi173 ceph-mon[25952]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:19.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph config set global log_to_journald false --force' 2024-06-21T01:49:20.564 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:20 smithi092 ceph-mon[25107]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:20.565 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:20 smithi092 ceph-mon[29810]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:20.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:20 smithi173 ceph-mon[25952]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:21.582 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:21 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:21.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:21 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:21.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:21 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:22.580 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:22 smithi092 ceph-mon[25107]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:22.580 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:22 smithi092 ceph-mon[29810]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:22.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-06-21T01:49:22.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:22 smithi173 ceph-mon[25952]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:22.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:49:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:49:23.791 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:49:23.791 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:49:23.791 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:49:23.791 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:49:23.791 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:49:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:23] "GET /metrics HTTP/1.1" 200 214938 "" "Prometheus/2.33.4" 2024-06-21T01:49:23.791 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:49:23.792 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:49:23.792 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:49:23.792 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:49:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:49:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:49:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:49:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:49:24.435 INFO:teuthology.orchestra.run.smithi092.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:49:24.780 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:24 smithi092 ceph-mon[29810]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:24.780 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:24.781 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:24 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:49:24.781 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:24 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:49:24.781 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:24 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:24.781 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:24 smithi092 ceph-mon[25107]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:24.781 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:24.781 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:24 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:49:24.781 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:24 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:49:24.781 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:24 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:24.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:24 smithi173 ceph-mon[25952]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:24.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:24.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:24 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:49:24.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:24 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:49:24.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:24 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:25.100 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-21T01:49:25.110 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi092.front.sepia.ceph.com 2024-06-21T01:49:25.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-06-21T01:49:25.700 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:25 smithi092 ceph-mon[25107]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:25.700 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:25 smithi092 ceph-mon[25107]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:49:25.700 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:25 smithi092 ceph-mon[25107]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:49:25.701 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:25 smithi092 ceph-mon[29810]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:25.701 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:25 smithi092 ceph-mon[29810]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:49:25.701 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:25 smithi092 ceph-mon[29810]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:49:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:25 smithi173 ceph-mon[25952]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:25 smithi173 ceph-mon[25952]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:49:25.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:25 smithi173 ceph-mon[25952]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:49:26.734 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:26 smithi092 ceph-mon[25107]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:26.735 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:26 smithi092 ceph-mon[29810]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:26 smithi173 ceph-mon[25952]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:27.208 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (110s) 34s ago 2m 16.3M - ba2b418f427c 510d745d0b55 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (2m) 34s ago 2m 53.7M - 8.3.5 dad864ee21e9 4c8715b5a8ac 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (118s) 34s ago 118s 46.3M - 3.5 e1d6a67b021e 6e77d5ec58b1 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443 running (7m) 34s ago 7m 411M - 17.2.0 e1d6a67b021e 8ba723b662bc 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:9283 running (8m) 34s ago 8m 457M - 17.2.0 e1d6a67b021e 0b61fab2e6e2 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (9m) 34s ago 9m 50.2M 2048M 17.2.0 e1d6a67b021e 3daa4a891e57 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (7m) 34s ago 7m 43.1M 2048M 17.2.0 e1d6a67b021e 00fcf1f13930 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (7m) 34s ago 7m 40.0M 2048M 17.2.0 e1d6a67b021e 6a24d6b0bbb6 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (3m) 34s ago 3m 16.0M - 1dbe0e931976 71773b5f07f0 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (3m) 34s ago 3m 17.6M - 1dbe0e931976 8455f1de1751 2024-06-21T01:49:27.990 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (7m) 34s ago 7m 50.7M 4016M 17.2.0 e1d6a67b021e 63507222842b 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (6m) 34s ago 6m 52.6M 4016M 17.2.0 e1d6a67b021e e0bf7c0600e5 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (6m) 34s ago 6m 46.0M 4016M 17.2.0 e1d6a67b021e 157fb16866c9 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (5m) 34s ago 5m 48.9M 4016M 17.2.0 e1d6a67b021e e5567e44eaec 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (5m) 34s ago 5m 52.7M 4272M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (5m) 34s ago 5m 51.2M 4272M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (4m) 34s ago 4m 47.4M 4272M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (4m) 34s ago 4m 50.1M 4272M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (105s) 34s ago 3m 51.5M - 514e6a882f6e c36e844319fc 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (2m) 34s ago 2m 73.0M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (2m) 34s ago 2m 69.5M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (2m) 34s ago 2m 70.4M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:49:27.991 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (2m) 34s ago 2m 70.9M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:49:28.487 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:49:28.487 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": {}, 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:49:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-06-21T01:49:28.489 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:49:28.489 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:49:28.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:28 smithi092 ceph-mon[25107]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:28.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:28 smithi092 ceph-mon[25107]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:28.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:28 smithi092 ceph-mon[25107]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:28.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:28 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1958597757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:49:28.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:28 smithi092 ceph-mon[29810]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:28.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:28 smithi092 ceph-mon[29810]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:28.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:28 smithi092 ceph-mon[29810]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:28.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:28 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1958597757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:49:28.915 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:49:28.916 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:49:28.916 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:49:28.916 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [], 2024-06-21T01:49:28.916 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "", 2024-06-21T01:49:28.916 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd image" 2024-06-21T01:49:28.916 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:49:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:28 smithi173 ceph-mon[25952]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:28 smithi173 ceph-mon[25952]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:28 smithi173 ceph-mon[25952]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:28.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:28 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1958597757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:49:29.390 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:49:29.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:29 smithi092 ceph-mon[29810]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:29.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:29 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1778537283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:49:29.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:29 smithi092 ceph-mon[25107]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:29.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:29 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1778537283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:49:29.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:29 smithi173 ceph-mon[25952]: from='client.24922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:29.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:29 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1778537283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:49:30.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:30 smithi092 ceph-mon[25107]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:30.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:30 smithi092 ceph-mon[25107]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:30.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:30 smithi092 ceph-mon[29810]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:30.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:30 smithi092 ceph-mon[29810]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:30 smithi173 ceph-mon[25952]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:30 smithi173 ceph-mon[25952]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:49:31.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:31 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:31.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:31 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:31.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:31 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:32.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:32 smithi173 ceph-mon[25952]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:32.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:49:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:49:32.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:32 smithi092 ceph-mon[25107]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:32.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:32 smithi092 ceph-mon[29810]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:34.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:49:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:33] "GET /metrics HTTP/1.1" 200 214938 "" "Prometheus/2.33.4" 2024-06-21T01:49:34.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:34 smithi092 ceph-mon[25107]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:34.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:34 smithi092 ceph-mon[29810]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:34.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:34 smithi173 ceph-mon[25952]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:36.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:36 smithi092 ceph-mon[25107]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:36.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:36 smithi092 ceph-mon[29810]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:36.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:36 smithi173 ceph-mon[25952]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:38 smithi092 ceph-mon[29810]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:38.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:38 smithi092 ceph-mon[25107]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:38.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:38 smithi173 ceph-mon[25952]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:40.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:40 smithi092 ceph-mon[25107]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:40.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:40 smithi092 ceph-mon[29810]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:40.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:40 smithi173 ceph-mon[25952]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:41.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:41 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:41.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:41 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:41.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:41 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:42.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:42 smithi173 ceph-mon[25952]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:42.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:49:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:49:42.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:42 smithi092 ceph-mon[29810]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:42.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:42 smithi092 ceph-mon[25107]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:44.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:49:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:43] "GET /metrics HTTP/1.1" 200 214948 "" "Prometheus/2.33.4" 2024-06-21T01:49:44.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:44 smithi092 ceph-mon[25107]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:44.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:44 smithi092 ceph-mon[29810]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:44.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:44 smithi173 ceph-mon[25952]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:46.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:46 smithi092 ceph-mon[25107]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:46.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:46 smithi092 ceph-mon[29810]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:46.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:46 smithi173 ceph-mon[25952]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:48.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:48 smithi092 ceph-mon[25107]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:48.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:48 smithi092 ceph-mon[29810]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:48.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:48 smithi173 ceph-mon[25952]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:50 smithi092 ceph-mon[25107]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:50.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:50 smithi092 ceph-mon[29810]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:50 smithi173 ceph-mon[25952]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:51 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:51.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:51 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:51.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:51 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:49:52.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:49:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:49:52.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:52 smithi092 ceph-mon[25107]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:52.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:52 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:52.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:52 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:49:52.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:52 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:52.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:52 smithi092 ceph-mon[29810]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:52.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:52 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:52.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:52 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:49:52.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:52 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:52.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:52 smithi173 ceph-mon[25952]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:52.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:52 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:52.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:52 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:49:52.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:52 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:49:53.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:49:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:49:53] "GET /metrics HTTP/1.1" 200 214909 "" "Prometheus/2.33.4" 2024-06-21T01:49:53.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:53 smithi092 ceph-mon[25107]: Upgrade: Target is version 19.0.0-2541-gc65be810 (unknown) 2024-06-21T01:49:53.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:53 smithi092 ceph-mon[25107]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, digests ['quay.ceph.io/ceph-ci/ceph@sha256:bba7e76a3887fd02e4f2e2ed3d352a62ae2630a800f64fb0e303a73f215a3fe0'] 2024-06-21T01:49:53.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:53 smithi092 ceph-mon[25107]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-21T01:49:53.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:53 smithi092 ceph-mon[25107]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd on smithi173 2024-06-21T01:49:53.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:53 smithi092 ceph-mon[29810]: Upgrade: Target is version 19.0.0-2541-gc65be810 (unknown) 2024-06-21T01:49:53.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:53 smithi092 ceph-mon[29810]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, digests ['quay.ceph.io/ceph-ci/ceph@sha256:bba7e76a3887fd02e4f2e2ed3d352a62ae2630a800f64fb0e303a73f215a3fe0'] 2024-06-21T01:49:53.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:53 smithi092 ceph-mon[29810]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-21T01:49:53.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:53 smithi092 ceph-mon[29810]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd on smithi173 2024-06-21T01:49:53.852 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:49:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:49:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:49:53.853 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:49:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:49:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-06-21T01:49:53.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:53 smithi173 ceph-mon[25952]: Upgrade: Target is version 19.0.0-2541-gc65be810 (unknown) 2024-06-21T01:49:53.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:53 smithi173 ceph-mon[25952]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, digests ['quay.ceph.io/ceph-ci/ceph@sha256:bba7e76a3887fd02e4f2e2ed3d352a62ae2630a800f64fb0e303a73f215a3fe0'] 2024-06-21T01:49:53.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:53 smithi173 ceph-mon[25952]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-21T01:49:53.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:53 smithi173 ceph-mon[25952]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd on smithi173 2024-06-21T01:49:54.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:54 smithi092 ceph-mon[25107]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:54.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:54 smithi092 ceph-mon[29810]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:54.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:54 smithi173 ceph-mon[25952]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:56.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:56 smithi092 ceph-mon[25107]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:56.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:56 smithi092 ceph-mon[29810]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:56.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:56 smithi173 ceph-mon[25952]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:49:58.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:49:58 smithi092 ceph-mon[25107]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:58.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:49:58 smithi092 ceph-mon[29810]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:58.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:49:58 smithi173 ceph-mon[25952]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:49:59.717 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:50:00.298 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:50:00.299 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (2m) 67s ago 3m 16.3M - ba2b418f427c 510d745d0b55 2024-06-21T01:50:00.299 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (2m) 67s ago 2m 53.7M - 8.3.5 dad864ee21e9 4c8715b5a8ac 2024-06-21T01:50:00.299 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (2m) 67s ago 2m 46.3M - 3.5 e1d6a67b021e 6e77d5ec58b1 2024-06-21T01:50:00.299 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443 running (7m) 67s ago 7m 411M - 17.2.0 e1d6a67b021e 8ba723b662bc 2024-06-21T01:50:00.299 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:9283 running (9m) 67s ago 9m 457M - 17.2.0 e1d6a67b021e 0b61fab2e6e2 2024-06-21T01:50:00.299 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (9m) 67s ago 9m 50.2M 2048M 17.2.0 e1d6a67b021e 3daa4a891e57 2024-06-21T01:50:00.299 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (8m) 67s ago 8m 43.1M 2048M 17.2.0 e1d6a67b021e 00fcf1f13930 2024-06-21T01:50:00.299 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (8m) 67s ago 8m 40.0M 2048M 17.2.0 e1d6a67b021e 6a24d6b0bbb6 2024-06-21T01:50:00.299 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (4m) 67s ago 4m 16.0M - 1dbe0e931976 71773b5f07f0 2024-06-21T01:50:00.299 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (4m) 67s ago 4m 17.6M - 1dbe0e931976 8455f1de1751 2024-06-21T01:50:00.299 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (7m) 67s ago 7m 50.7M 4016M 17.2.0 e1d6a67b021e 63507222842b 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (7m) 67s ago 7m 52.6M 4016M 17.2.0 e1d6a67b021e e0bf7c0600e5 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (6m) 67s ago 6m 46.0M 4016M 17.2.0 e1d6a67b021e 157fb16866c9 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (6m) 67s ago 6m 48.9M 4016M 17.2.0 e1d6a67b021e e5567e44eaec 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (6m) 67s ago 6m 52.7M 4272M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (5m) 67s ago 5m 51.2M 4272M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (5m) 67s ago 5m 47.4M 4272M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (4m) 67s ago 4m 50.1M 4272M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (2m) 67s ago 4m 51.5M - 514e6a882f6e c36e844319fc 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (2m) 67s ago 2m 73.0M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (2m) 67s ago 2m 69.5M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (2m) 67s ago 2m 70.4M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:50:00.300 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (2m) 67s ago 2m 70.9M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:50:00.648 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:50:00.648 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:50:00.648 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-06-21T01:50:00.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:50:00.648 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:50:00.648 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": {}, 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:50:00.649 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:50:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:00 smithi092 ceph-mon[25107]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:00 smithi092 ceph-mon[25107]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:00.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:00 smithi092 ceph-mon[25107]: overall HEALTH_OK 2024-06-21T01:50:00.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:00 smithi092 ceph-mon[29810]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:00.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:00 smithi092 ceph-mon[29810]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:00.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:00 smithi092 ceph-mon[29810]: overall HEALTH_OK 2024-06-21T01:50:00.944 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:50:00.944 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:50:00.944 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:50:00.944 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [], 2024-06-21T01:50:00.944 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "0/23 daemons upgraded", 2024-06-21T01:50:00.944 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd image on host smithi173" 2024-06-21T01:50:00.944 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:50:00.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:00 smithi173 ceph-mon[25952]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:00.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:00 smithi173 ceph-mon[25952]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:00.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:00 smithi173 ceph-mon[25952]: overall HEALTH_OK 2024-06-21T01:50:01.293 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:50:01.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:01 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:01.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:01 smithi092 ceph-mon[29810]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:01.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:01 smithi092 ceph-mon[29810]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:01.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:01 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/310648399' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:50:01.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:01 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3831711008' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:50:01.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:01 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:01.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:01 smithi092 ceph-mon[25107]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:01.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:01 smithi092 ceph-mon[25107]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:01.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:01 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/310648399' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:50:01.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:01 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3831711008' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:50:01.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:01 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:01.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:01 smithi173 ceph-mon[25952]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:01.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:01 smithi173 ceph-mon[25952]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:01.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:01 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/310648399' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:50:01.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:01 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3831711008' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:50:02.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:02 smithi173 ceph-mon[25952]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:02.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:02 smithi173 ceph-mon[25952]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:02.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:50:02.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:02 smithi092 ceph-mon[25107]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:02.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:02 smithi092 ceph-mon[25107]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:02.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:02 smithi092 ceph-mon[29810]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:02.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:02 smithi092 ceph-mon[29810]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:03.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:03] "GET /metrics HTTP/1.1" 200 214909 "" "Prometheus/2.33.4" 2024-06-21T01:50:03.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=error ts=2024-06-21T01:50:03.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:50:03.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:50:03.852 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-06-21T01:50:04.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:04 smithi173 ceph-mon[25952]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:05.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:04 smithi092 ceph-mon[25107]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:05.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:04 smithi092 ceph-mon[29810]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:06.903 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:06 smithi092 ceph-mon[29810]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:06.903 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:06 smithi092 ceph-mon[25107]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:06.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:06 smithi173 ceph-mon[25952]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:08.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:08 smithi173 ceph-mon[25952]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:09.060 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:08 smithi092 ceph-mon[25107]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:09.060 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:08 smithi092 ceph-mon[29810]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:10.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:10 smithi173 ceph-mon[25952]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:11.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:10 smithi092 ceph-mon[25107]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:11.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:10 smithi092 ceph-mon[29810]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:11.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:11 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:12.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:11 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:12.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:11 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:12.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:12 smithi173 ceph-mon[25952]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:12.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:50:13.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:12 smithi092 ceph-mon[25107]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:13.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:12 smithi092 ceph-mon[29810]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:13.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:13 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:13] "GET /metrics HTTP/1.1" 200 214892 "" "Prometheus/2.33.4" 2024-06-21T01:50:13.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:13 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=error ts=2024-06-21T01:50:13.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:50:13.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:13 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-06-21T01:50:13.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:13 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:50:14.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:14 smithi173 ceph-mon[25952]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:15.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:14 smithi092 ceph-mon[25107]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:15.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:14 smithi092 ceph-mon[29810]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:16.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:16 smithi173 ceph-mon[25952]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:17.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:16 smithi092 ceph-mon[25107]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:17.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:16 smithi092 ceph-mon[29810]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:18.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:18 smithi092 ceph-mon[25107]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:18.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:18 smithi092 ceph-mon[29810]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:18.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:18 smithi173 ceph-mon[25952]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:20.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:20 smithi173 ceph-mon[25952]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:21.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:20 smithi092 ceph-mon[25107]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:21.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:20 smithi092 ceph-mon[29810]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:21.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:21 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:22.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:21 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:22.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:21 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:22.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[27129]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:50:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:22 smithi173 ceph-mon[25952]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:22 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:22 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:50:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:22 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:50:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:22 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:50:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:22 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:50:23.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[29810]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:23.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:23.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:50:23.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:50:23.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:50:23.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:50:23.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[25107]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:23.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:23.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:50:23.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:50:23.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:50:23.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:22 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:50:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[25107]: Upgrade: Updating mgr.x 2024-06-21T01:50:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[25107]: Deploying daemon mgr.x on smithi173 2024-06-21T01:50:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:23.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:23.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:23] "GET /metrics HTTP/1.1" 200 214894 "" "Prometheus/2.33.4" 2024-06-21T01:50:23.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[29810]: Upgrade: Updating mgr.x 2024-06-21T01:50:23.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[29810]: Deploying daemon mgr.x on smithi173 2024-06-21T01:50:23.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:23.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:23.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:23.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:23.853 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=error ts=2024-06-21T01:50:23.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:50:23.853 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-06-21T01:50:23.853 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:50:23.909 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:23 smithi173 ceph-mon[25952]: Upgrade: Updating mgr.x 2024-06-21T01:50:23.909 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:23 smithi173 ceph-mon[25952]: Deploying daemon mgr.x on smithi173 2024-06-21T01:50:23.909 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:23.909 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:23 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:23.909 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:23.909 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:23 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:24.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:24 smithi173 systemd[1]: Stopping Ceph mgr.x for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:50:24.229 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:24 smithi173 bash[46858]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr.x" found: no such container 2024-06-21T01:50:24.586 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:24 smithi173 podman[46869]: 2024-06-21 01:50:24.295324181 +0000 UTC m=+0.144569993 container died 8ba723b662bcd4df1a845f7beca94040761c279e4d998f79b74ea6c1307dab58 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, version=8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, name=centos-stream, architecture=x86_64, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-06-21T01:50:24.840 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:24 smithi173 ceph-mon[25952]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:24.841 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:24 smithi173 podman[46869]: 2024-06-21 01:50:24.590777582 +0000 UTC m=+0.440023393 container cleanup 8ba723b662bcd4df1a845f7beca94040761c279e4d998f79b74ea6c1307dab58 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, ceph=True, com.redhat.component=centos-stream-container, version=8, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-06-21T01:50:24.841 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:24 smithi173 bash[46869]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x 2024-06-21T01:50:24.841 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:24 smithi173 bash[46898]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr.x" found: no such container 2024-06-21T01:50:24.841 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:24 smithi173 podman[46882]: 2024-06-21 01:50:24.825669702 +0000 UTC m=+0.526455878 container remove 8ba723b662bcd4df1a845f7beca94040761c279e4d998f79b74ea6c1307dab58 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-type=git, architecture=x86_64, vendor=Red Hat, Inc.) 2024-06-21T01:50:24.841 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:24 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-06-21T01:50:24.841 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:24 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-06-21T01:50:24.841 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:24 smithi173 systemd[1]: Stopped Ceph mgr.x for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:50:24.841 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:24 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.x.service: Consumed 16.651s CPU time. 2024-06-21T01:50:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:24 smithi092 ceph-mon[25107]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:25.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:24 smithi092 ceph-mon[29810]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:25.101 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:24.789Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.173:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-06-21T01:50:25.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:25 smithi173 systemd[1]: Starting Ceph mgr.x for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:50:25.534 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:25 smithi173 podman[47000]: 2024-06-21 01:50:25.153379639 +0000 UTC m=+0.017508423 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:50:25.534 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:25 smithi173 podman[47000]: 2024-06-21 01:50:25.251959608 +0000 UTC m=+0.116088433 container create d64fa854065ae1386f499bf5474120a0892abb0e833eed0e54a61eea232cd091 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618) 2024-06-21T01:50:25.534 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:25 smithi173 podman[47000]: 2024-06-21 01:50:25.471056128 +0000 UTC m=+0.335184902 container init d64fa854065ae1386f499bf5474120a0892abb0e833eed0e54a61eea232cd091 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2) 2024-06-21T01:50:25.534 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:25 smithi173 podman[47000]: 2024-06-21 01:50:25.473794307 +0000 UTC m=+0.337923076 container start d64fa854065ae1386f499bf5474120a0892abb0e833eed0e54a61eea232cd091 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-06-21T01:50:25.534 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:25 smithi173 ceph-mgr[47018]: -- 172.21.15.173:0/362440549 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5635a280d0e0 con 0x5635a27eb400 2024-06-21T01:50:25.848 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:25 smithi173 bash[47000]: d64fa854065ae1386f499bf5474120a0892abb0e833eed0e54a61eea232cd091 2024-06-21T01:50:25.848 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:25 smithi173 systemd[1]: Started Ceph mgr.x for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:50:25.848 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:25 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:25.606+0000 7fbc810b1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:50:25.848 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:25 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:25.769+0000 7fbc810b1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:50:25.848 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:25 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:25.851+0000 7fbc810b1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:50:26.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:26 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:26.116+0000 7fbc810b1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:50:26.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:26 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:26.182+0000 7fbc810b1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:50:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:26 smithi092 ceph-mon[25107]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:26 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:26 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:50:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:26 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:50:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:26 smithi092 ceph-mon[29810]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:26 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:26 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:50:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:26 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:50:26.924 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:26 smithi173 ceph-mon[25952]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:26.924 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:26 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:26.925 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:26 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:50:26.925 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:26 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:50:27.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:27.196+0000 7fbc810b1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:50:27.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:27.328+0000 7fbc810b1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:50:27.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:27.392+0000 7fbc810b1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:50:27.903 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:27.622+0000 7fbc810b1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:50:27.903 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:27.689+0000 7fbc810b1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:50:27.903 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:27.762+0000 7fbc810b1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:50:28.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:27.916+0000 7fbc810b1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:50:28.704 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:28 smithi173 ceph-mon[25952]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:28.705 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:28 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:28.366+0000 7fbc810b1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:50:28.705 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:28 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:28.707+0000 7fbc810b1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:50:28.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:28 smithi092 ceph-mon[25107]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:28.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:28 smithi092 ceph-mon[29810]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:28.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:28 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:28.777+0000 7fbc810b1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:50:28.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:28 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:28.944+0000 7fbc810b1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-21T01:50:29.348 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:29 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:29.286+0000 7fbc810b1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:50:29.695 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:29 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:29.351+0000 7fbc810b1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:50:29.696 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:29 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:29.541+0000 7fbc810b1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:50:29.696 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:29 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:29.698+0000 7fbc810b1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:50:29.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:29 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:29.772+0000 7fbc810b1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:50:29.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:29 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:29.864+0000 7fbc810b1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:50:29.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:29 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:29.928+0000 7fbc810b1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:50:30.683 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:30.518+0000 7fbc810b1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:50:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-mon[25952]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-mon[25952]: Standby manager daemon x restarted 2024-06-21T01:50:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-mon[25952]: Standby manager daemon x started 2024-06-21T01:50:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:50:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:50:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:50:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:50:30.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:30.696+0000 7fbc810b1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:50:30.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:50:30] ENGINE Bus STARTING 2024-06-21T01:50:30.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: CherryPy Checker: 2024-06-21T01:50:30.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: The Application mounted at '' has an empty config. 2024-06-21T01:50:30.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:30.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:50:30] ENGINE Serving on http://:::9283 2024-06-21T01:50:30.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:30 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:50:30] ENGINE Bus STARTED 2024-06-21T01:50:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[25107]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[25107]: Standby manager daemon x restarted 2024-06-21T01:50:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[25107]: Standby manager daemon x started 2024-06-21T01:50:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[29810]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[29810]: Standby manager daemon x restarted 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[29810]: Standby manager daemon x started 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:50:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:30 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:50:31.617 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:50:31.857 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:31 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:31.857 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:31 smithi173 ceph-mon[25952]: mgrmap e21: y(active, since 5m), standbys: x 2024-06-21T01:50:32.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:31 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:32.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:31 smithi092 ceph-mon[25107]: mgrmap e21: y(active, since 5m), standbys: x 2024-06-21T01:50:32.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:31 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:32.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:31 smithi092 ceph-mon[29810]: mgrmap e21: y(active, since 5m), standbys: x 2024-06-21T01:50:32.215 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (2m) 99s ago 3m 16.3M - ba2b418f427c 510d745d0b55 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (3m) 2s ago 3m 53.9M - 8.3.5 dad864ee21e9 4c8715b5a8ac 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (3m) 99s ago 3m 46.3M - 3.5 e1d6a67b021e 6e77d5ec58b1 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283 running (7s) 2s ago 8m 49.4M - 19.0.0-2541-gc65be810 e9bfc9e474e5 d64fa854065a 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:9283 running (10m) 99s ago 10m 457M - 17.2.0 e1d6a67b021e 0b61fab2e6e2 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (10m) 99s ago 10m 50.2M 2048M 17.2.0 e1d6a67b021e 3daa4a891e57 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (8m) 2s ago 8m 43.3M 2048M 17.2.0 e1d6a67b021e 00fcf1f13930 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (8m) 99s ago 8m 40.0M 2048M 17.2.0 e1d6a67b021e 6a24d6b0bbb6 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (4m) 99s ago 4m 16.0M - 1dbe0e931976 71773b5f07f0 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (4m) 2s ago 4m 19.7M - 1dbe0e931976 8455f1de1751 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (8m) 99s ago 8m 50.7M 4016M 17.2.0 e1d6a67b021e 63507222842b 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (7m) 99s ago 7m 52.6M 4016M 17.2.0 e1d6a67b021e e0bf7c0600e5 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (7m) 99s ago 7m 46.0M 4016M 17.2.0 e1d6a67b021e 157fb16866c9 2024-06-21T01:50:32.216 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (6m) 99s ago 6m 48.9M 4016M 17.2.0 e1d6a67b021e e5567e44eaec 2024-06-21T01:50:32.217 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (6m) 2s ago 6m 53.7M 4272M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:50:32.217 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (6m) 2s ago 6m 52.0M 4272M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:50:32.217 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (5m) 2s ago 5m 48.0M 4272M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:50:32.217 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (5m) 2s ago 5m 50.8M 4272M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:50:32.217 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (2m) 2s ago 4m 65.1M - 514e6a882f6e c36e844319fc 2024-06-21T01:50:32.217 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (3m) 99s ago 3m 73.0M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:50:32.217 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (3m) 2s ago 3m 70.2M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:50:32.217 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (3m) 99s ago 3m 70.4M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:50:32.217 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (3m) 2s ago 3m 71.5M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:50:32.568 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 1 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": {}, 2024-06-21T01:50:32.569 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:50:32.570 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:50:32.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:50:32.570 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:50:32.570 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-06-21T01:50:32.570 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 1 2024-06-21T01:50:32.570 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:50:32.570 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:50:32.649 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:50:32.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:32 smithi092 ceph-mon[25107]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:32.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:32 smithi092 ceph-mon[25107]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:32.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:32 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3356813819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:50:32.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:32 smithi092 ceph-mon[29810]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:32.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:32 smithi092 ceph-mon[29810]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:32.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:32 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3356813819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:50:32.869 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:50:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:50:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:50:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [], 2024-06-21T01:50:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "1/23 daemons upgraded", 2024-06-21T01:50:32.870 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading mgr daemons" 2024-06-21T01:50:32.870 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:50:32.960 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:32 smithi173 ceph-mon[25952]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:32.960 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:32 smithi173 ceph-mon[25952]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:32.960 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:32 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3356813819' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:50:33.216 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:50:33.506 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:33.139Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:50:33.783 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:33] "GET /metrics HTTP/1.1" 200 214894 "" "Prometheus/2.33.4" 2024-06-21T01:50:33.783 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=error ts=2024-06-21T01:50:33.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:50:33.784 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:33.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-06-21T01:50:33.784 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:33.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:50:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[25107]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[25107]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3702533090' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:50:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:50:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[25107]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-21T01:50:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-21T01:50:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[25107]: osdmap e97: 8 total, 8 up, 8 in 2024-06-21T01:50:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[29810]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[29810]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3702533090' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:50:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:50:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[29810]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-21T01:50:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-21T01:50:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:33 smithi092 ceph-mon[29810]: osdmap e97: 8 total, 8 up, 8 in 2024-06-21T01:50:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:33 smithi173 ceph-mon[25952]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:33 smithi173 ceph-mon[25952]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:33 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3702533090' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:50:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:33 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:33 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' 2024-06-21T01:50:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:33 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:50:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:33 smithi173 ceph-mon[25952]: from='mgr.14448 172.21.15.92:0/872394916' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-21T01:50:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:33 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-21T01:50:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:33 smithi173 ceph-mon[25952]: osdmap e97: 8 total, 8 up, 8 in 2024-06-21T01:50:34.599 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:34.304+0000 7fb7b69f2700 -1 mgr handle_mgr_map I was active but no longer am 2024-06-21T01:50:34.599 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ignoring --setuser ceph since I am not root 2024-06-21T01:50:34.600 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ignoring --setgroup ceph since I am not root 2024-06-21T01:50:34.600 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mgr[25340]: -- 172.21.15.92:0/2155325936 <== mon.0 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561d31d7a340 con 0x561d31d52800 2024-06-21T01:50:34.600 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:34.474+0000 7fe3d934d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:50:34.600 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:34.602+0000 7fe3d934d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:50:34.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:50:34] ENGINE Bus STOPPING 2024-06-21T01:50:34.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:50:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-21T01:50:34.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:50:34] ENGINE Bus STOPPED 2024-06-21T01:50:34.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:34.700+0000 7fe3d934d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:50:34.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='client.25006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:34.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:34.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-21T01:50:34.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: Failing over to other MGR 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: mgrmap e22: x(active, starting, since 0.996948s) 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:50:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: Manager daemon x is now available 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='client.25006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: Failing over to other MGR 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: mgrmap e22: x(active, starting, since 0.996948s) 2024-06-21T01:50:34.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:50:34.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:50:34.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:50:34.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: Manager daemon x is now available 2024-06-21T01:50:34.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:34.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:34.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:35.150 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:50:34] ENGINE Bus STARTING 2024-06-21T01:50:35.150 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:50:34] ENGINE Serving on http://:::9283 2024-06-21T01:50:35.151 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:50:34] ENGINE Bus STARTED 2024-06-21T01:50:35.151 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='client.25006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:50:35.151 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:35.151 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-21T01:50:35.151 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: Failing over to other MGR 2024-06-21T01:50:35.152 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-06-21T01:50:35.152 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: mgrmap e22: x(active, starting, since 0.996948s) 2024-06-21T01:50:35.152 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:50:35.152 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:50:35.152 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:50:35.152 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:50:35.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:50:35.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:50:35.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:50:35.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:50:35.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:50:35.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:50:35.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:50:35.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:50:35.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:50:35.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:50:35.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:50:35.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: Manager daemon x is now available 2024-06-21T01:50:35.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:35.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:35.155 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:34 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:35.351 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:34.996Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-06-21T01:50:35.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:35.015+0000 7fe3d934d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:50:35.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:35.098+0000 7fe3d934d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:50:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: Queued rgw.foo for migration 2024-06-21T01:50:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: Queued rgw.smpl for migration 2024-06-21T01:50:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-06-21T01:50:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-06-21T01:50:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:50:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:36.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:36.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:36.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:36.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: mgrmap e23: x(active, since 2s) 2024-06-21T01:50:36.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:36.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: Deploying cephadm binary to smithi173 2024-06-21T01:50:36.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: Deploying cephadm binary to smithi092 2024-06-21T01:50:36.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:36.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: Queued rgw.foo for migration 2024-06-21T01:50:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: Queued rgw.smpl for migration 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: mgrmap e23: x(active, since 2s) 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: Deploying cephadm binary to smithi173 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: Deploying cephadm binary to smithi092 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:36.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:35 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:36.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: Queued rgw.foo for migration 2024-06-21T01:50:36.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: Queued rgw.smpl for migration 2024-06-21T01:50:36.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: mgrmap e23: x(active, since 2s) 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: Deploying cephadm binary to smithi173 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: Deploying cephadm binary to smithi092 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:36.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:35 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:36.414 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:36 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:36.240+0000 7fe3d934d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:50:36.415 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:36 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:36.417+0000 7fe3d934d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:50:36.839 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:36 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:36.509+0000 7fe3d934d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:50:36.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:36.676+0000 7fbbd8b61640 -1 log_channel(cephadm) log [ERR] : [21/Jun/2024:01:50:36] ENGINE Error in HTTPServer.serve 2024-06-21T01:50:36.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Traceback (most recent call last): 2024-06-21T01:50:36.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:50:36.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: self._connections.run(self.expiration_interval) 2024-06-21T01:50:36.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:50:36.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: self._run(expiration_interval) 2024-06-21T01:50:36.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:50:36.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:50:36.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:50:36.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:50:36.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:50:36.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: s = self.context.wrap_socket( 2024-06-21T01:50:36.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:50:36.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return self.sslsocket_class._create( 2024-06-21T01:50:36.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:50:36.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: self.do_handshake() 2024-06-21T01:50:36.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:50:36.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: self._sslobj.do_handshake() 2024-06-21T01:50:36.980 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:50:36.981 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:50:37.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:36 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:36.841+0000 7fe3d934d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:50:37.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:36 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:36.936+0000 7fe3d934d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:50:37.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:37 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:37.035+0000 7fe3d934d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:50:37.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:37 smithi092 ceph-mon[25107]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:37.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:37 smithi092 ceph-mon[25107]: [21/Jun/2024:01:50:36] ENGINE Bus STARTING 2024-06-21T01:50:37.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:37 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:37.309+0000 7fe3d934d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:50:37.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:37 smithi092 ceph-mon[29810]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:37.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:37 smithi092 ceph-mon[29810]: [21/Jun/2024:01:50:36] ENGINE Bus STARTING 2024-06-21T01:50:37.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:37 smithi173 ceph-mon[25952]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:37.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:37 smithi173 ceph-mon[25952]: [21/Jun/2024:01:50:36] ENGINE Bus STARTING 2024-06-21T01:50:38.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:37 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:37.787+0000 7fe3d934d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:50:38.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:38.336+0000 7fe3d934d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:50:38.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:38.439+0000 7fe3d934d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:50:38.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-mon[25107]: [21/Jun/2024:01:50:36] ENGINE Serving on http://172.21.15.173:8765 2024-06-21T01:50:38.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-mon[25107]: [21/Jun/2024:01:50:36] ENGINE Serving on https://172.21.15.173:7150 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-mon[25107]: [21/Jun/2024:01:50:36] ENGINE Bus STARTED 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-mon[25107]: [21/Jun/2024:01:50:36] ENGINE Error in HTTPServer.serve 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout: self._connections.run(self.expiration_interval) 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout: self._run(expiration_interval) 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:50:38.602 INFO:journalctl@ceph.mon.a.smithi092.stdout: s = self.context.wrap_socket( 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.a.smithi092.stdout: return self.sslsocket_class._create( 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.a.smithi092.stdout: self.do_handshake() 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.a.smithi092.stdout: self._sslobj.do_handshake() 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.a.smithi092.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-mon[25107]: mgrmap e24: x(active, since 4s) 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-mon[29810]: [21/Jun/2024:01:50:36] ENGINE Serving on http://172.21.15.173:8765 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-mon[29810]: [21/Jun/2024:01:50:36] ENGINE Serving on https://172.21.15.173:7150 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-mon[29810]: [21/Jun/2024:01:50:36] ENGINE Bus STARTED 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-mon[29810]: [21/Jun/2024:01:50:36] ENGINE Error in HTTPServer.serve 2024-06-21T01:50:38.603 INFO:journalctl@ceph.mon.c.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: self._connections.run(self.expiration_interval) 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: self._run(expiration_interval) 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: s = self.context.wrap_socket( 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: return self.sslsocket_class._create( 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:50:38.604 INFO:journalctl@ceph.mon.c.smithi092.stdout: self.do_handshake() 2024-06-21T01:50:38.605 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:50:38.605 INFO:journalctl@ceph.mon.c.smithi092.stdout: self._sslobj.do_handshake() 2024-06-21T01:50:38.605 INFO:journalctl@ceph.mon.c.smithi092.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:50:38.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-mon[29810]: mgrmap e24: x(active, since 4s) 2024-06-21T01:50:38.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:38 smithi173 ceph-mon[25952]: [21/Jun/2024:01:50:36] ENGINE Serving on http://172.21.15.173:8765 2024-06-21T01:50:38.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:38 smithi173 ceph-mon[25952]: [21/Jun/2024:01:50:36] ENGINE Serving on https://172.21.15.173:7150 2024-06-21T01:50:38.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:38 smithi173 ceph-mon[25952]: [21/Jun/2024:01:50:36] ENGINE Bus STARTED 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:38 smithi173 ceph-mon[25952]: [21/Jun/2024:01:50:36] ENGINE Error in HTTPServer.serve 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: Traceback (most recent call last): 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._connections.run(self.expiration_interval) 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._run(expiration_interval) 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: s = self.context.wrap_socket( 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:50:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout: return self.sslsocket_class._create( 2024-06-21T01:50:38.980 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:50:38.980 INFO:journalctl@ceph.mon.b.smithi173.stdout: self.do_handshake() 2024-06-21T01:50:38.980 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:50:38.980 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._sslobj.do_handshake() 2024-06-21T01:50:38.980 INFO:journalctl@ceph.mon.b.smithi173.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:50:38.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:38 smithi173 ceph-mon[25952]: mgrmap e24: x(active, since 4s) 2024-06-21T01:50:39.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:38.745+0000 7fe3d934d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:50:39.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:38.837+0000 7fe3d934d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:50:39.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:39 smithi092 ceph-mon[25107]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:39.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:39 smithi092 ceph-mon[29810]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:39.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:39.260+0000 7fe3d934d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:50:39.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:39.397+0000 7fe3d934d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:50:39.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:39.493+0000 7fe3d934d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:50:39.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:39 smithi173 ceph-mon[25952]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:39.935 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:39.611+0000 7fe3d934d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:50:39.935 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:39.707+0000 7fe3d934d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:50:40.276 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:40.278+0000 7fe3d934d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:50:40.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-mon[25107]: mgrmap e25: x(active, since 6s) 2024-06-21T01:50:40.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:40.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:40.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-mon[29810]: mgrmap e25: x(active, since 6s) 2024-06-21T01:50:40.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:40.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:40.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:40 smithi173 ceph-mon[25952]: mgrmap e25: x(active, since 6s) 2024-06-21T01:50:40.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:40 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:40.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:40 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:41.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:40.659+0000 7fe3d934d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:50:41.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:50:40] ENGINE Bus STARTING 2024-06-21T01:50:41.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: CherryPy Checker: 2024-06-21T01:50:41.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: The Application mounted at '' has an empty config. 2024-06-21T01:50:41.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: 2024-06-21T01:50:41.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:50:40] ENGINE Serving on http://:::9283 2024-06-21T01:50:41.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: [21/Jun/2024:01:50:40] ENGINE Bus STARTED 2024-06-21T01:50:41.494 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:41.175Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-06-21T01:50:41.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: Standby manager daemon y started 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: Standby manager daemon y started 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-21T01:50:41.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:50:41.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:41.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:41.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:41 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:41.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: Standby manager daemon y started 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.92:0/1659918172' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:41.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:41 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:42.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:42 smithi173 ceph-mon[25952]: Adjusting osd_memory_target on smithi092 to 2736M 2024-06-21T01:50:42.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:42 smithi173 ceph-mon[25952]: mgrmap e26: x(active, since 8s), standbys: y 2024-06-21T01:50:42.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:42 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:50:42.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:42 smithi173 ceph-mon[25952]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:50:42.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:42 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:42] "GET /metrics HTTP/1.1" 200 35148 "" "Prometheus/2.33.4" 2024-06-21T01:50:42.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:42 smithi092 ceph-mon[25107]: Adjusting osd_memory_target on smithi092 to 2736M 2024-06-21T01:50:42.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:42 smithi092 ceph-mon[25107]: mgrmap e26: x(active, since 8s), standbys: y 2024-06-21T01:50:42.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:42 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:50:42.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:42 smithi092 ceph-mon[25107]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:50:42.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:42 smithi092 ceph-mon[29810]: Adjusting osd_memory_target on smithi092 to 2736M 2024-06-21T01:50:42.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:42 smithi092 ceph-mon[29810]: mgrmap e26: x(active, since 8s), standbys: y 2024-06-21T01:50:42.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:42 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:50:42.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:42 smithi092 ceph-mon[29810]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:50:43.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:50:43.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=error ts=2024-06-21T01:50:43.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-06-21T01:50:43.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:43.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-06-21T01:50:43.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:43.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:50:45.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:45 smithi173 ceph-mon[25952]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-21T01:50:45.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:45 smithi092 ceph-mon[25107]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-21T01:50:45.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:45 smithi092 ceph-mon[29810]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-21T01:50:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:47 smithi092 ceph-mon[25107]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-21T01:50:47.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:47 smithi092 ceph-mon[29810]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-21T01:50:47.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:47 smithi173 ceph-mon[25952]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-21T01:50:49.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:49 smithi092 ceph-mon[25107]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:50:49.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:49 smithi092 ceph-mon[29810]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:50:49.929 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:49 smithi173 ceph-mon[25952]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:50:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:50 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:50:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:50 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:50.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:50 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:50 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:50:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:50 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:50 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:50 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:50:50.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:50 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:50.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:50 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:51.718 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:51 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:51.718 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:51 smithi173 ceph-mon[25952]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:50:51.718 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:51 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:51.718 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:51 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:51 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:51 smithi092 ceph-mon[25107]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:50:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:51 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:51 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:51.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:51 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:50:51.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:51 smithi092 ceph-mon[29810]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:50:51.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:51 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:51.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:51 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:52.686 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:50:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:52] "GET /metrics HTTP/1.1" 200 38152 "" "Prometheus/2.33.4" 2024-06-21T01:50:53.726 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:53 smithi173 ceph-mon[25952]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:50:53.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:53 smithi092 ceph-mon[25107]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:50:53.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:53 smithi092 ceph-mon[29810]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:50:53.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:50:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:50:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:50:53.852 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=error ts=2024-06-21T01:50:53.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-06-21T01:50:53.852 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:53.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-06-21T01:50:53.852 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:50:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=warn ts=2024-06-21T01:50:53.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.173:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.173 because it doesn't contain any IP SANs" 2024-06-21T01:50:55.420 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.421 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.421 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.421 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.421 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.421 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.421 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.421 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.421 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.421 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.421 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: Adjusting osd_memory_target on smithi173 to 2992M 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.422 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.423 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.423 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.423 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.423 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.423 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.423 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.423 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.423 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.423 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.423 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.424 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.424 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: Adjusting osd_memory_target on smithi173 to 2992M 2024-06-21T01:50:55.424 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.424 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:50:55.424 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:50:55.424 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:50:55.424 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:50:55.424 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:55.424 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:50:55.424 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:50:55.424 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.425 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.425 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.425 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.425 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:55 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: Adjusting osd_memory_target on smithi173 to 2992M 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:55 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:55.851 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:55 smithi092 systemd[1]: Stopping Ceph node-exporter.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:50:55.851 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:55 smithi092 bash[58472]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter.a" found: no such container 2024-06-21T01:50:55.851 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:55 smithi092 podman[58484]: 2024-06-21 01:50:55.60359093 +0000 UTC m=+0.144924716 container died 71773b5f07f016cef897ab4d8cbf8dcf8c6e99a553a0d712112c79a398849116 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T01:50:56.351 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 podman[58484]: 2024-06-21 01:50:56.07603384 +0000 UTC m=+0.617367625 container cleanup 71773b5f07f016cef897ab4d8cbf8dcf8c6e99a553a0d712112c79a398849116 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T01:50:56.351 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 bash[58484]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a 2024-06-21T01:50:56.351 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 bash[58509]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter.a" found: no such container 2024-06-21T01:50:56.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[25107]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:50:56.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[25107]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:50:56.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[25107]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:50:56.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[25107]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:50:56.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[25107]: Reconfiguring node-exporter.a (dependencies changed)... 2024-06-21T01:50:56.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[25107]: Reconfiguring daemon node-exporter.a on smithi092 2024-06-21T01:50:56.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[29810]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:50:56.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[29810]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:50:56.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[29810]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:50:56.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[29810]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:50:56.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[29810]: Reconfiguring node-exporter.a (dependencies changed)... 2024-06-21T01:50:56.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-mon[29810]: Reconfiguring daemon node-exporter.a on smithi092 2024-06-21T01:50:56.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:56 smithi173 ceph-mon[25952]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:50:56.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:56 smithi173 ceph-mon[25952]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:50:56.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:56 smithi173 ceph-mon[25952]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:50:56.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:56 smithi173 ceph-mon[25952]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:50:56.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:56 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:56 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:56 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:56 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:56.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:56 smithi173 ceph-mon[25952]: Reconfiguring node-exporter.a (dependencies changed)... 2024-06-21T01:50:56.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:56 smithi173 ceph-mon[25952]: Reconfiguring daemon node-exporter.a on smithi092 2024-06-21T01:50:56.616 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 podman[58497]: 2024-06-21 01:50:56.361041216 +0000 UTC m=+0.761031688 container remove 71773b5f07f016cef897ab4d8cbf8dcf8c6e99a553a0d712112c79a398849116 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T01:50:56.616 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-06-21T01:50:56.616 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-06-21T01:50:56.616 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 systemd[1]: Stopped Ceph node-exporter.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:50:56.617 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.a.service: Consumed 2.399s CPU time. 2024-06-21T01:50:56.617 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 systemd[1]: Starting Ceph node-exporter.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:50:56.617 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 podman[58574]: 2024-06-21 01:50:56.493823607 +0000 UTC m=+0.017763355 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-06-21T01:50:56.617 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 podman[58574]: 2024-06-21 01:50:56.620424999 +0000 UTC m=+0.144364742 container create 099deaf4585360d46cc25fd5c6f76be6c453d08794386d277e08bcc479406836 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T01:50:56.902 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 podman[58574]: 2024-06-21 01:50:56.822478545 +0000 UTC m=+0.346418302 container init 099deaf4585360d46cc25fd5c6f76be6c453d08794386d277e08bcc479406836 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T01:50:56.902 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 podman[58574]: 2024-06-21 01:50:56.825278405 +0000 UTC m=+0.349218158 container start 099deaf4585360d46cc25fd5c6f76be6c453d08794386d277e08bcc479406836 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.828Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.828Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.828Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.828Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=arp 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=bcache 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=bonding 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=btrfs 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=conntrack 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=cpu 2024-06-21T01:50:56.903 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-06-21T01:50:56.904 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=diskstats 2024-06-21T01:50:56.904 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=dmi 2024-06-21T01:50:56.904 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=edac 2024-06-21T01:50:56.904 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=entropy 2024-06-21T01:50:56.904 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-06-21T01:50:56.904 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=filefd 2024-06-21T01:50:56.904 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=filesystem 2024-06-21T01:50:56.904 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=hwmon 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=infiniband 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=ipvs 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=loadavg 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=mdadm 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=meminfo 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=netclass 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=netdev 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=netstat 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=nfs 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=nfsd 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=nvme 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=os 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-06-21T01:50:56.905 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=pressure 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=rapl 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=schedstat 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=sockstat 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=softnet 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=stat 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.829Z caller=node_exporter.go:115 level=info collector=tapestats 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.830Z caller=node_exporter.go:115 level=info collector=textfile 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.830Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.830Z caller=node_exporter.go:115 level=info collector=time 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.830Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.830Z caller=node_exporter.go:115 level=info collector=uname 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.830Z caller=node_exporter.go:115 level=info collector=vmstat 2024-06-21T01:50:56.906 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.830Z caller=node_exporter.go:115 level=info collector=xfs 2024-06-21T01:50:56.907 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.830Z caller=node_exporter.go:115 level=info collector=zfs 2024-06-21T01:50:56.907 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.830Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-06-21T01:50:56.907 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a[58589]: ts=2024-06-21T01:50:56.830Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-06-21T01:50:56.907 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 bash[58574]: 099deaf4585360d46cc25fd5c6f76be6c453d08794386d277e08bcc479406836 2024-06-21T01:50:57.311 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 01:50:56 smithi092 systemd[1]: Started Ceph node-exporter.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:50:57.594 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:57 smithi092 ceph-mon[25107]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:57.594 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:57 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:57.594 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:57 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:57.594 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:57 smithi092 ceph-mon[29810]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:57.594 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:57 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:57.594 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:57 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:57.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:57 smithi173 ceph-mon[25952]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:50:57.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:57 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:57.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:57 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:50:58.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:58 smithi092 ceph-mon[25107]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-21T01:50:58.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:58 smithi092 ceph-mon[25107]: Reconfiguring daemon alertmanager.a on smithi092 2024-06-21T01:50:58.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:58 smithi092 ceph-mon[29810]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-21T01:50:58.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:58 smithi092 ceph-mon[29810]: Reconfiguring daemon alertmanager.a on smithi092 2024-06-21T01:50:58.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:58 smithi173 ceph-mon[25952]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-21T01:50:58.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:58 smithi173 ceph-mon[25952]: Reconfiguring daemon alertmanager.a on smithi092 2024-06-21T01:50:59.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:50:59 smithi092 ceph-mon[25107]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:59.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:50:59 smithi092 ceph-mon[29810]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:50:59.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:50:59 smithi173 ceph-mon[25952]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:00.338 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:00 smithi092 systemd[1]: Stopping Ceph alertmanager.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:00.338 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:00 smithi092 bash[58761]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager.a" found: no such container 2024-06-21T01:51:00.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:00 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[53421]: level=info ts=2024-06-21T01:51:00.342Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-06-21T01:51:00.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:00 smithi092 podman[58772]: 2024-06-21 01:51:00.38339088 +0000 UTC m=+0.135291848 container died 510d745d0b554cb82c22197bce7f5486ec2e76a49ac1234c3dadbe088a8093a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:01.099 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:00 smithi092 podman[58772]: 2024-06-21 01:51:00.842894982 +0000 UTC m=+0.594795989 container cleanup 510d745d0b554cb82c22197bce7f5486ec2e76a49ac1234c3dadbe088a8093a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:01.099 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:00 smithi092 bash[58772]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a 2024-06-21T01:51:01.099 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:00 smithi092 bash[58796]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager.a" found: no such container 2024-06-21T01:51:01.351 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 podman[58784]: 2024-06-21 01:51:01.102309041 +0000 UTC m=+0.722060572 container remove 510d745d0b554cb82c22197bce7f5486ec2e76a49ac1234c3dadbe088a8093a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:01.351 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 podman[58784]: 2024-06-21 01:51:01.211425795 +0000 UTC m=+0.831177326 volume remove 05f6843a2454d4d550abc508badf9820914e81ed64d2abd6421a51f2b073a3d7 2024-06-21T01:51:01.351 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-06-21T01:51:01.351 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 systemd[1]: Stopped Ceph alertmanager.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:01.351 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 systemd[1]: Starting Ceph alertmanager.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:01.799 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-mon[25107]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:51:01.799 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-mon[25107]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:01.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-mon[29810]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:51:01.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-mon[29810]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:01.799 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 podman[58858]: 2024-06-21 01:51:01.359940574 +0000 UTC m=+0.018201403 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-06-21T01:51:01.800 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 podman[58858]: 2024-06-21 01:51:01.487347221 +0000 UTC m=+0.145608094 volume create d91a3846db2d687ba1a0c2cbb1f01fdb2f0df7450cda0e1cdec23dbf7cd9897c 2024-06-21T01:51:01.800 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 podman[58858]: 2024-06-21 01:51:01.587870837 +0000 UTC m=+0.246131706 container create 9b53017582564c2343a251ead09c1c1aa111d04b9025ce042ef71a9c89945684 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:01.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:01 smithi173 ceph-mon[25952]: from='client.24700 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:51:01.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:01 smithi173 ceph-mon[25952]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:02.101 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 podman[58858]: 2024-06-21 01:51:01.90735537 +0000 UTC m=+0.565616198 container init 9b53017582564c2343a251ead09c1c1aa111d04b9025ce042ef71a9c89945684 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:02.101 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 podman[58858]: 2024-06-21 01:51:01.910300282 +0000 UTC m=+0.568561110 container start 9b53017582564c2343a251ead09c1c1aa111d04b9025ce042ef71a9c89945684 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:02.101 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=info ts=2024-06-21T01:51:01.920Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-06-21T01:51:02.102 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=info ts=2024-06-21T01:51:01.920Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-06-21T01:51:02.102 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=info ts=2024-06-21T01:51:01.921Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.92 port=9094 2024-06-21T01:51:02.102 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=info ts=2024-06-21T01:51:01.922Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-06-21T01:51:02.102 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=info ts=2024-06-21T01:51:01.947Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-21T01:51:02.102 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=info ts=2024-06-21T01:51:01.947Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-21T01:51:02.102 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=info ts=2024-06-21T01:51:01.949Z caller=main.go:518 msg=Listening address=:9093 2024-06-21T01:51:02.102 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=info ts=2024-06-21T01:51:01.950Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-06-21T01:51:02.102 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:02 smithi092 bash[58858]: 9b53017582564c2343a251ead09c1c1aa111d04b9025ce042ef71a9c89945684 2024-06-21T01:51:02.102 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:02 smithi092 systemd[1]: Started Ceph alertmanager.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:02.728 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:02.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.92:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.173:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:51:02.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:51:02] "GET /metrics HTTP/1.1" 200 38152 "" "Prometheus/2.33.4" 2024-06-21T01:51:03.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:03.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:03.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[29810]: Reconfiguring iscsi.foo.smithi092.pqnhsu (dependencies changed)... 2024-06-21T01:51:03.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:03.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:03.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:03.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[29810]: Reconfiguring daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:51:03.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[29810]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:51:03.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:03.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:03.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[25107]: Reconfiguring iscsi.foo.smithi092.pqnhsu (dependencies changed)... 2024-06-21T01:51:03.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:03.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:03.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:03.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[25107]: Reconfiguring daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:51:03.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-mon[25107]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:51:03.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:03 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:03.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:03 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:03.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:03 smithi173 ceph-mon[25952]: Reconfiguring iscsi.foo.smithi092.pqnhsu (dependencies changed)... 2024-06-21T01:51:03.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:03 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:03.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:03 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:03.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:03 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:03.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:03 smithi173 ceph-mon[25952]: Reconfiguring daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:51:03.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:03 smithi173 ceph-mon[25952]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:51:03.564 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:51:03.815 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[25336]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:51:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:51:04.101 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=info ts=2024-06-21T01:51:03.923Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000563079s 2024-06-21T01:51:04.231 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:51:04.231 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 starting - - - - 2024-06-21T01:51:04.231 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (3m) 13s ago 3m 53.9M - 8.3.5 dad864ee21e9 4c8715b5a8ac 2024-06-21T01:51:04.231 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (3m) 23s ago 3m 49.1M - 3.5 e1d6a67b021e 6e77d5ec58b1 2024-06-21T01:51:04.231 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283 running (39s) 13s ago 8m 483M - 19.0.0-2541-gc65be810 e9bfc9e474e5 d64fa854065a 2024-06-21T01:51:04.231 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:9283 running (10m) 23s ago 10m 212M - 17.2.0 e1d6a67b021e 0b61fab2e6e2 2024-06-21T01:51:04.231 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (10m) 23s ago 10m 54.2M 2048M 17.2.0 e1d6a67b021e 3daa4a891e57 2024-06-21T01:51:04.231 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (9m) 13s ago 9m 45.8M 2048M 17.2.0 e1d6a67b021e 00fcf1f13930 2024-06-21T01:51:04.231 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (9m) 23s ago 9m 42.4M 2048M 17.2.0 e1d6a67b021e 6a24d6b0bbb6 2024-06-21T01:51:04.231 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 starting - - - - 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (5m) 13s ago 5m 18.6M - 1.3.1 1dbe0e931976 8455f1de1751 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (8m) 23s ago 8m 52.1M 2736M 17.2.0 e1d6a67b021e 63507222842b 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (8m) 23s ago 8m 54.0M 2736M 17.2.0 e1d6a67b021e e0bf7c0600e5 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (7m) 23s ago 7m 46.9M 2736M 17.2.0 e1d6a67b021e 157fb16866c9 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (7m) 23s ago 7m 50.2M 2736M 17.2.0 e1d6a67b021e e5567e44eaec 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (7m) 13s ago 7m 54.2M 2992M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (6m) 13s ago 6m 52.2M 2992M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (6m) 13s ago 6m 48.6M 2992M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (6m) 13s ago 6m 51.4M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (3m) 13s ago 5m 65.1M - 2.33.4 514e6a882f6e c36e844319fc 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (3m) 23s ago 3m 74.2M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (3m) 13s ago 3m 70.9M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (3m) 23s ago 3m 71.7M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:51:04.232 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (3m) 13s ago 3m 72.2M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:51:04.638 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:51:04.638 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:51:04.639 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-06-21T01:51:04.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:51:04.639 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:51:04.639 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-06-21T01:51:04.639 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 1 2024-06-21T01:51:04.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:51:04.640 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:51:04.640 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-21T01:51:04.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:51:04.640 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": {}, 2024-06-21T01:51:04.640 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:51:04.640 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:51:04.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:51:04.641 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:51:04.641 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-06-21T01:51:04.641 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 1 2024-06-21T01:51:04.641 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:51:04.641 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:51:04.974 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:51:04.974 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:51:04.974 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:51:04.974 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:51:04.974 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [], 2024-06-21T01:51:04.974 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "1/23 daemons upgraded", 2024-06-21T01:51:04.974 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "", 2024-06-21T01:51:04.974 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:51:04.974 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:51:05.375 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:51:05.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[29810]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:05.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[29810]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:05.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[29810]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:05.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[29810]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:05.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2852084256' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:51:05.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:05.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[25107]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:05.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[25107]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:05.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[25107]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:05.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[25107]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:05.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2852084256' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:51:05.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:05 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:05.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-mon[25952]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:05.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-mon[25952]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:05.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-mon[25952]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:05.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-mon[25952]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:05.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2852084256' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:51:05.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:05.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:05.447+0000 7fbc1160f640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T01:51:05.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Creating ceph-iscsi config... 2024-06-21T01:51:05.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T01:51:05.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T01:51:05.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Non-zero exit code 1 from systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T01:51:05.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Traceback (most recent call last): 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return _run_code(code, main_globals, None, 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: exec(code, run_globals) 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1123, in deploy_daemon 2024-06-21T01:51:05.730 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: RuntimeError: Failed command: systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Traceback (most recent call last): 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: self.mgr._daemon_action(daemon_spec, action=action) 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2456, in _daemon_action 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return self.wait_async( 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return self.event_loop.get_result(coro, timeout) 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return future.result(timeout) 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return self.__get_result() 2024-06-21T01:51:05.731 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: raise self._exception 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1350, in _create_daemon 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: out, err, code = await self._run_cephadm( 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1691, in _run_cephadm 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: raise OrchestratorError( 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Creating ceph-iscsi config... 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Non-zero exit code 1 from systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Traceback (most recent call last): 2024-06-21T01:51:05.732 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return _run_code(code, main_globals, None, 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: exec(code, run_globals) 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1123, in deploy_daemon 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: RuntimeError: Failed command: systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:05.733 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:06 smithi173 ceph-mon[25952]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:06.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:06 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2141454665' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:51:06.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:06 smithi173 ceph-mon[25952]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T01:51:06.729 INFO:journalctl@ceph.mon.b.smithi173.stdout: Creating ceph-iscsi config... 2024-06-21T01:51:06.730 INFO:journalctl@ceph.mon.b.smithi173.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T01:51:06.730 INFO:journalctl@ceph.mon.b.smithi173.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T01:51:06.730 INFO:journalctl@ceph.mon.b.smithi173.stdout: Non-zero exit code 1 from systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T01:51:06.730 INFO:journalctl@ceph.mon.b.smithi173.stdout: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.730 INFO:journalctl@ceph.mon.b.smithi173.stdout: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.731 INFO:journalctl@ceph.mon.b.smithi173.stdout: Traceback (most recent call last): 2024-06-21T01:51:06.731 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T01:51:06.731 INFO:journalctl@ceph.mon.b.smithi173.stdout: return _run_code(code, main_globals, None, 2024-06-21T01:51:06.731 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T01:51:06.731 INFO:journalctl@ceph.mon.b.smithi173.stdout: exec(code, run_globals) 2024-06-21T01:51:06.731 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T01:51:06.732 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T01:51:06.732 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T01:51:06.732 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T01:51:06.732 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T01:51:06.732 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1123, in deploy_daemon 2024-06-21T01:51:06.733 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T01:51:06.733 INFO:journalctl@ceph.mon.b.smithi173.stdout: RuntimeError: Failed command: systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.733 INFO:journalctl@ceph.mon.b.smithi173.stdout: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.733 INFO:journalctl@ceph.mon.b.smithi173.stdout: Traceback (most recent call last): 2024-06-21T01:51:06.733 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-06-21T01:51:06.733 INFO:journalctl@ceph.mon.b.smithi173.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-06-21T01:51:06.734 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2456, in _daemon_action 2024-06-21T01:51:06.734 INFO:journalctl@ceph.mon.b.smithi173.stdout: return self.wait_async( 2024-06-21T01:51:06.734 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-06-21T01:51:06.734 INFO:journalctl@ceph.mon.b.smithi173.stdout: return self.event_loop.get_result(coro, timeout) 2024-06-21T01:51:06.734 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-06-21T01:51:06.734 INFO:journalctl@ceph.mon.b.smithi173.stdout: return future.result(timeout) 2024-06-21T01:51:06.735 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-06-21T01:51:06.735 INFO:journalctl@ceph.mon.b.smithi173.stdout: return self.__get_result() 2024-06-21T01:51:06.735 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-06-21T01:51:06.735 INFO:journalctl@ceph.mon.b.smithi173.stdout: raise self._exception 2024-06-21T01:51:06.735 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1350, in _create_daemon 2024-06-21T01:51:06.735 INFO:journalctl@ceph.mon.b.smithi173.stdout: out, err, code = await self._run_cephadm( 2024-06-21T01:51:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1691, in _run_cephadm 2024-06-21T01:51:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout: raise OrchestratorError( 2024-06-21T01:51:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T01:51:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout: Creating ceph-iscsi config... 2024-06-21T01:51:06.736 INFO:journalctl@ceph.mon.b.smithi173.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T01:51:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T01:51:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout: Non-zero exit code 1 from systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T01:51:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout: Traceback (most recent call last): 2024-06-21T01:51:06.737 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T01:51:06.738 INFO:journalctl@ceph.mon.b.smithi173.stdout: return _run_code(code, main_globals, None, 2024-06-21T01:51:06.738 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T01:51:06.738 INFO:journalctl@ceph.mon.b.smithi173.stdout: exec(code, run_globals) 2024-06-21T01:51:06.738 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T01:51:06.738 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T01:51:06.738 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T01:51:06.738 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T01:51:06.739 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T01:51:06.739 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1123, in deploy_daemon 2024-06-21T01:51:06.739 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T01:51:06.739 INFO:journalctl@ceph.mon.b.smithi173.stdout: RuntimeError: Failed command: systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.739 INFO:journalctl@ceph.mon.b.smithi173.stdout: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:06 smithi173 ceph-mon[25952]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-21T01:51:06.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:06 smithi092 ceph-mon[29810]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:06.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:06 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2141454665' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:51:06.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:06 smithi092 ceph-mon[29810]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T01:51:06.851 INFO:journalctl@ceph.mon.c.smithi092.stdout: Creating ceph-iscsi config... 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: Non-zero exit code 1 from systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: return _run_code(code, main_globals, None, 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: exec(code, run_globals) 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T01:51:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1123, in deploy_daemon 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: RuntimeError: Failed command: systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2456, in _daemon_action 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: return self.wait_async( 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: return self.event_loop.get_result(coro, timeout) 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-06-21T01:51:06.853 INFO:journalctl@ceph.mon.c.smithi092.stdout: return future.result(timeout) 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: return self.__get_result() 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: raise self._exception 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1350, in _create_daemon 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: out, err, code = await self._run_cephadm( 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1691, in _run_cephadm 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: raise OrchestratorError( 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: Creating ceph-iscsi config... 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: Non-zero exit code 1 from systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T01:51:06.854 INFO:journalctl@ceph.mon.c.smithi092.stdout: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: return _run_code(code, main_globals, None, 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: exec(code, run_globals) 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1123, in deploy_daemon 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: RuntimeError: Failed command: systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.855 INFO:journalctl@ceph.mon.c.smithi092.stdout: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.856 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:06 smithi092 ceph-mon[29810]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-21T01:51:06.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:06 smithi092 ceph-mon[25107]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:06.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:06 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2141454665' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:51:06.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:06 smithi092 ceph-mon[25107]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T01:51:06.856 INFO:journalctl@ceph.mon.a.smithi092.stdout: Creating ceph-iscsi config... 2024-06-21T01:51:06.856 INFO:journalctl@ceph.mon.a.smithi092.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T01:51:06.856 INFO:journalctl@ceph.mon.a.smithi092.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T01:51:06.856 INFO:journalctl@ceph.mon.a.smithi092.stdout: Non-zero exit code 1 from systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T01:51:06.856 INFO:journalctl@ceph.mon.a.smithi092.stdout: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.856 INFO:journalctl@ceph.mon.a.smithi092.stdout: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.857 INFO:journalctl@ceph.mon.a.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:51:06.857 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T01:51:06.857 INFO:journalctl@ceph.mon.a.smithi092.stdout: return _run_code(code, main_globals, None, 2024-06-21T01:51:06.857 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T01:51:06.857 INFO:journalctl@ceph.mon.a.smithi092.stdout: exec(code, run_globals) 2024-06-21T01:51:06.857 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T01:51:06.857 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T01:51:06.857 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T01:51:06.857 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1123, in deploy_daemon 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: RuntimeError: Failed command: systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2456, in _daemon_action 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: return self.wait_async( 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: return self.event_loop.get_result(coro, timeout) 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-06-21T01:51:06.858 INFO:journalctl@ceph.mon.a.smithi092.stdout: return future.result(timeout) 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: return self.__get_result() 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: raise self._exception 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1350, in _create_daemon 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: out, err, code = await self._run_cephadm( 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1691, in _run_cephadm 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: raise OrchestratorError( 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: Creating ceph-iscsi config... 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: Non-zero exit code 1 from systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.859 INFO:journalctl@ceph.mon.a.smithi092.stdout: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: return _run_code(code, main_globals, None, 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: exec(code, run_globals) 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1123, in deploy_daemon 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: RuntimeError: Failed command: systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:06 smithi092 ceph-mon[25107]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-21T01:51:07.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:07 smithi173 ceph-mon[25952]: Reconfiguring daemon prometheus.a on smithi173 2024-06-21T01:51:07.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:07 smithi173 ceph-mon[25952]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:51:07.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:07 smithi092 ceph-mon[25107]: Reconfiguring daemon prometheus.a on smithi173 2024-06-21T01:51:07.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:07 smithi092 ceph-mon[25107]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:51:07.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:07 smithi092 ceph-mon[29810]: Reconfiguring daemon prometheus.a on smithi173 2024-06-21T01:51:07.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:07 smithi092 ceph-mon[29810]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:51:08.395 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 systemd[1]: Stopping Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:08.395 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 bash[50284]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus.a" found: no such container 2024-06-21T01:51:08.687 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.398Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-21T01:51:08.687 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.398Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-06-21T01:51:08.687 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.398Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-06-21T01:51:08.687 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.398Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-06-21T01:51:08.687 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.398Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-06-21T01:51:08.687 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.398Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-06-21T01:51:08.687 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.398Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-21T01:51:08.687 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.398Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-06-21T01:51:08.687 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.398Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-06-21T01:51:08.688 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.399Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-06-21T01:51:08.688 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.399Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-06-21T01:51:08.688 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[45000]: ts=2024-06-21T01:51:08.399Z caller=main.go:1066 level=info msg="See you next time!" 2024-06-21T01:51:08.688 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 podman[50296]: 2024-06-21 01:51:08.408632699 +0000 UTC m=+0.129971762 container died c36e844319fcfb19a2bb3674c9b8fa8f181b5bb8231d37105043b53b50d2467e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:08.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 podman[50296]: 2024-06-21 01:51:08.789835486 +0000 UTC m=+0.511174548 container cleanup c36e844319fcfb19a2bb3674c9b8fa8f181b5bb8231d37105043b53b50d2467e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:08.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 bash[50296]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a 2024-06-21T01:51:08.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:08 smithi173 bash[50321]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus.a" found: no such container 2024-06-21T01:51:09.305 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 podman[50307]: 2024-06-21 01:51:09.016079996 +0000 UTC m=+0.608286255 container remove c36e844319fcfb19a2bb3674c9b8fa8f181b5bb8231d37105043b53b50d2467e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:09.305 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-06-21T01:51:09.305 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 systemd[1]: Stopped Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:09.305 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@prometheus.a.service: Consumed 1.271s CPU time. 2024-06-21T01:51:09.305 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 systemd[1]: Starting Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:09.305 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 podman[50383]: 2024-06-21 01:51:09.155456438 +0000 UTC m=+0.018662265 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-06-21T01:51:09.305 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 podman[50383]: 2024-06-21 01:51:09.309020504 +0000 UTC m=+0.172226324 container create 77bcbddbb5ac896c05a81a67a669be6cf3a5ad0caed7bc9502954d84a323c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:09.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-mon[25952]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:51:09.565 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 podman[50383]: 2024-06-21 01:51:09.477778419 +0000 UTC m=+0.340984242 container init 77bcbddbb5ac896c05a81a67a669be6cf3a5ad0caed7bc9502954d84a323c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:09.565 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 podman[50383]: 2024-06-21 01:51:09.480336866 +0000 UTC m=+0.343542688 container start 77bcbddbb5ac896c05a81a67a669be6cf3a5ad0caed7bc9502954d84a323c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:09.566 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.505Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-06-21T01:51:09.566 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.505Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-06-21T01:51:09.566 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.505Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-06-21T01:51:09.566 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.505Z caller=main.go:518 level=info host_details="(Linux 5.14.0-460.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024 x86_64 smithi173 (none))" 2024-06-21T01:51:09.566 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.505Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-06-21T01:51:09.566 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.505Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-06-21T01:51:09.566 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.507Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-06-21T01:51:09.566 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.507Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-06-21T01:51:09.567 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.508Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-06-21T01:51:09.567 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.511Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-06-21T01:51:09.567 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.511Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.009µs 2024-06-21T01:51:09.567 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.511Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-06-21T01:51:09.567 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 bash[50383]: 77bcbddbb5ac896c05a81a67a669be6cf3a5ad0caed7bc9502954d84a323c24e 2024-06-21T01:51:09.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:09 smithi092 ceph-mon[25107]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:51:09.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:09 smithi092 ceph-mon[29810]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:51:09.887 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 systemd[1]: Started Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:09.888 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.725Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-06-21T01:51:10.204 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:09 smithi173 systemd[1]: Stopping Ceph node-exporter.b for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:10.204 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:09 smithi173 bash[50505]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter.b" found: no such container 2024-06-21T01:51:10.204 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 podman[50516]: 2024-06-21 01:51:10.033254937 +0000 UTC m=+0.109874226 container died 8455f1de1751d1702c53fa53d0a685676c537d0c0b708e3619e2346d2579a11e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T01:51:10.204 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.938Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-06-21T01:51:10.204 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.938Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-06-21T01:51:10.204 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.938Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=29.219µs wal_replay_duration=426.645942ms total_replay_duration=426.688986ms 2024-06-21T01:51:10.204 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.941Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-06-21T01:51:10.205 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.941Z caller=main.go:947 level=info msg="TSDB started" 2024-06-21T01:51:10.205 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.941Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-06-21T01:51:10.205 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.965Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.478049ms db_storage=800ns remote_storage=1.719µs web_handler=413ns query_engine=981ns scrape=452.109µs scrape_sd=150.149µs notify=18.885µs notify_sd=27.411µs rules=22.371891ms 2024-06-21T01:51:10.205 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:09.965Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-06-21T01:51:10.478 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 podman[50516]: 2024-06-21 01:51:10.356141111 +0000 UTC m=+0.432760399 container cleanup 8455f1de1751d1702c53fa53d0a685676c537d0c0b708e3619e2346d2579a11e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T01:51:10.478 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 bash[50516]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b 2024-06-21T01:51:10.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:10 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:10.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:10 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:10.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:10 smithi173 ceph-mon[25952]: Reconfiguring node-exporter.b (dependencies changed)... 2024-06-21T01:51:10.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:10 smithi173 ceph-mon[25952]: Reconfiguring daemon node-exporter.b on smithi173 2024-06-21T01:51:10.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:10 smithi173 ceph-mon[25952]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:51:10.830 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 podman[50530]: 2024-06-21 01:51:10.574367823 +0000 UTC m=+0.545284533 container remove 8455f1de1751d1702c53fa53d0a685676c537d0c0b708e3619e2346d2579a11e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T01:51:10.830 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-06-21T01:51:10.830 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 bash[50543]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter.b" found: no such container 2024-06-21T01:51:10.830 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-06-21T01:51:10.831 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 systemd[1]: Stopped Ceph node-exporter.b for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:10.831 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.b.service: Consumed 2.445s CPU time. 2024-06-21T01:51:10.831 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 systemd[1]: Starting Ceph node-exporter.b for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:10.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:10 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:10.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:10 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:10.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:10 smithi092 ceph-mon[25107]: Reconfiguring node-exporter.b (dependencies changed)... 2024-06-21T01:51:10.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:10 smithi092 ceph-mon[25107]: Reconfiguring daemon node-exporter.b on smithi173 2024-06-21T01:51:10.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:10 smithi092 ceph-mon[25107]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:51:10.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:10 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:10.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:10 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:10.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:10 smithi092 ceph-mon[29810]: Reconfiguring node-exporter.b (dependencies changed)... 2024-06-21T01:51:10.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:10 smithi092 ceph-mon[29810]: Reconfiguring daemon node-exporter.b on smithi173 2024-06-21T01:51:10.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:10 smithi092 ceph-mon[29810]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:51:11.124 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 podman[50612]: 2024-06-21 01:51:10.833745938 +0000 UTC m=+0.113869149 container create e0cf70cfc7524c2f5bbd6b6faae31b8cb6cd895fc43c73053e05887369e42593 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T01:51:11.124 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:10 smithi173 podman[50612]: 2024-06-21 01:51:10.737541928 +0000 UTC m=+0.017665146 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-06-21T01:51:11.124 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 podman[50612]: 2024-06-21 01:51:11.035787586 +0000 UTC m=+0.315910807 container init e0cf70cfc7524c2f5bbd6b6faae31b8cb6cd895fc43c73053e05887369e42593 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T01:51:11.124 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 podman[50612]: 2024-06-21 01:51:11.039244608 +0000 UTC m=+0.319367820 container start e0cf70cfc7524c2f5bbd6b6faae31b8cb6cd895fc43c73053e05887369e42593 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T01:51:11.124 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.042Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-06-21T01:51:11.124 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.042Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-06-21T01:51:11.124 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.042Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-06-21T01:51:11.124 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.042Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-21T01:51:11.125 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-06-21T01:51:11.125 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=arp 2024-06-21T01:51:11.125 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=bcache 2024-06-21T01:51:11.125 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=bonding 2024-06-21T01:51:11.125 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=btrfs 2024-06-21T01:51:11.125 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=conntrack 2024-06-21T01:51:11.125 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=cpu 2024-06-21T01:51:11.125 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=diskstats 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=dmi 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=edac 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=entropy 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=filefd 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=filesystem 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=hwmon 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=infiniband 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=ipvs 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=loadavg 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=mdadm 2024-06-21T01:51:11.126 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=meminfo 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=netclass 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=netdev 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=netstat 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=nfs 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=nfsd 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=nvme 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=os 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=pressure 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=rapl 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=schedstat 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=sockstat 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=softnet 2024-06-21T01:51:11.127 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=stat 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=tapestats 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=textfile 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=time 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=uname 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=vmstat 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=xfs 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:115 level=info collector=zfs 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b[50629]: ts=2024-06-21T01:51:11.043Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-06-21T01:51:11.128 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 bash[50612]: e0cf70cfc7524c2f5bbd6b6faae31b8cb6cd895fc43c73053e05887369e42593 2024-06-21T01:51:11.383 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 01:51:11 smithi173 systemd[1]: Started Ceph node-exporter.b for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:12.351 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 01:51:11 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=info ts=2024-06-21T01:51:11.926Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003555175s 2024-06-21T01:51:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[25107]: Reconfiguring grafana.a (dependencies changed)... 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[25107]: Regenerating cephadm self-signed grafana TLS certificates 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[25107]: Reconfiguring daemon grafana.a on smithi173 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[29810]: Reconfiguring grafana.a (dependencies changed)... 2024-06-21T01:51:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[29810]: Regenerating cephadm self-signed grafana TLS certificates 2024-06-21T01:51:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:51:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:51:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:12 smithi092 ceph-mon[29810]: Reconfiguring daemon grafana.a on smithi173 2024-06-21T01:51:12.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:12 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:12 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:12 smithi173 ceph-mon[25952]: Reconfiguring grafana.a (dependencies changed)... 2024-06-21T01:51:12.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:12 smithi173 ceph-mon[25952]: Regenerating cephadm self-signed grafana TLS certificates 2024-06-21T01:51:12.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:12 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:12 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:12.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:12 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:51:12.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:12 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-21T01:51:12.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:12 smithi173 ceph-mon[25952]: Reconfiguring daemon grafana.a on smithi173 2024-06-21T01:51:13.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:13 smithi173 ceph-mon[25952]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:51:13.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:13 smithi092 ceph-mon[25107]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:51:13.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:13 smithi092 ceph-mon[29810]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:51:14.228 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:13 smithi173 systemd[1]: Stopping Ceph grafana.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:14.228 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:13 smithi173 bash[50799]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana.a" found: no such container 2024-06-21T01:51:14.229 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[41953]: t=2024-06-21T01:51:14+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-06-21T01:51:14.229 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 podman[50810]: 2024-06-21 01:51:14.064151672 +0000 UTC m=+0.116227916 container died 4c8715b5a8aced021ee6d1c4229199bbf1faf3b183278fbcdf53a971f685600f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , description=Ceph Grafana Container, name=ubi8, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, release=236.1648460182, version=8.5, distribution-scope=public) 2024-06-21T01:51:14.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 podman[50810]: 2024-06-21 01:51:14.412506815 +0000 UTC m=+0.464583103 container cleanup 4c8715b5a8aced021ee6d1c4229199bbf1faf3b183278fbcdf53a971f685600f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a, release=236.1648460182, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, name=ubi8, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-06-21T01:51:14.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 bash[50810]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a 2024-06-21T01:51:14.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 podman[50823]: 2024-06-21 01:51:14.563950395 +0000 UTC m=+0.493514745 container remove 4c8715b5a8aced021ee6d1c4229199bbf1faf3b183278fbcdf53a971f685600f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, description=Ceph Grafana Container, name=ubi8, com.redhat.component=ubi8-container, io.openshift.expose-services=, vendor=Red Hat, Inc., io.buildah.version=1.24.2, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public) 2024-06-21T01:51:14.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 bash[50839]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana.a" found: no such container 2024-06-21T01:51:14.903 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@grafana.a.service: Deactivated successfully. 2024-06-21T01:51:14.903 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 systemd[1]: Stopped Ceph grafana.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:14.903 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@grafana.a.service: Consumed 2.246s CPU time. 2024-06-21T01:51:14.903 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 systemd[1]: Starting Ceph grafana.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:14.903 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 podman[50900]: 2024-06-21 01:51:14.751686804 +0000 UTC m=+0.016355877 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-06-21T01:51:14.903 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:14 smithi173 podman[50900]: 2024-06-21 01:51:14.906807957 +0000 UTC m=+0.171477017 container create 0003e9fdc5d3cb5a3bb9c6557e732af0a99327ac3a4d2f1a377a4ba9cc04371f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, maintainer=Paul Cuzner , release=236.1648460182, name=ubi8, io.openshift.tags=base rhel8, version=8.5, vendor=Red Hat, Inc.) 2024-06-21T01:51:15.161 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 podman[50900]: 2024-06-21 01:51:15.134218288 +0000 UTC m=+0.398887365 container init 0003e9fdc5d3cb5a3bb9c6557e732af0a99327ac3a4d2f1a377a4ba9cc04371f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, name=ubi8, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, io.openshift.tags=base rhel8, architecture=x86_64, vcs-type=git, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-06-21T01:51:15.161 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 podman[50900]: 2024-06-21 01:51:15.137030101 +0000 UTC m=+0.401699163 container start 0003e9fdc5d3cb5a3bb9c6557e732af0a99327ac3a4d2f1a377a4ba9cc04371f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a, maintainer=Paul Cuzner , release=236.1648460182, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, vcs-type=git, distribution-scope=public, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, io.openshift.expose-services=, version=8.5, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2) 2024-06-21T01:51:15.424 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:15] ENGINE Bus STOPPING 2024-06-21T01:51:15.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-21T01:51:15.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:51:15.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-06-21T01:51:15.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:15.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-06-21T01:51:15.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:51:15.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-06-21T01:51:15.426 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.426 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:51:15.426 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-06-21T01:51:15.426 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-06-21T01:51:15.426 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-06-21T01:51:15.426 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-06-21T01:51:15.426 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-06-21T01:51:15.426 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-06-21T01:51:15.426 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="App mode production" logger=settings 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=501.964µs 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-06-21T01:51:15.427 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-06-21T01:51:15.428 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-06-21T01:51:15.428 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-06-21T01:51:15.428 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-06-21T01:51:15.428 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 bash[50900]: 0003e9fdc5d3cb5a3bb9c6557e732af0a99327ac3a4d2f1a377a4ba9cc04371f 2024-06-21T01:51:15.428 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 systemd[1]: Started Ceph grafana.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:15.428 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-06-21T01:51:15.428 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-06-21T01:51:15.428 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-06-21T01:51:15.428 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-06-21T01:51:15.428 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-06-21T01:51:15.428 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T01:51:15+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-06-21T01:51:15.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-21T01:51:15.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:15] ENGINE Bus STOPPED 2024-06-21T01:51:15.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:15] ENGINE Bus STARTING 2024-06-21T01:51:15.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:15] ENGINE Serving on http://:::9283 2024-06-21T01:51:15.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:15] ENGINE Bus STARTED 2024-06-21T01:51:15.729 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:15] ENGINE Bus STOPPING 2024-06-21T01:51:15.751 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.752 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:51:15.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-21T01:51:15.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:51:15.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-06-21T01:51:15.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:15.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-06-21T01:51:15.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:51:15.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-06-21T01:51:15.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:15.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:15 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:51:16.381 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-21T01:51:16.381 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:16] ENGINE Bus STOPPED 2024-06-21T01:51:16.381 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:16] ENGINE Bus STARTING 2024-06-21T01:51:16.382 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:16] ENGINE Serving on http://:::9283 2024-06-21T01:51:16.382 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:16] ENGINE Bus STARTED 2024-06-21T01:51:16.382 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:16] ENGINE Bus STOPPING 2024-06-21T01:51:16.641 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:51:16.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-06-21T01:51:16.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:16.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-06-21T01:51:16.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:51:16.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-06-21T01:51:16.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-mon[25952]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-21T01:51:16.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-mon[25952]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-21T01:51:16.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:51:16.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-06-21T01:51:16.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:16.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-06-21T01:51:16.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:51:16.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-06-21T01:51:16.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[25107]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-21T01:51:16.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[25107]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-21T01:51:16.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-21T01:51:16.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-06-21T01:51:16.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:16.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-06-21T01:51:16.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-21T01:51:16.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-06-21T01:51:16.728 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[29810]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-21T01:51:16.728 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:16 smithi092 ceph-mon[29810]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-21T01:51:16.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-21T01:51:16.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:16] ENGINE Bus STOPPED 2024-06-21T01:51:16.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:16] ENGINE Bus STARTING 2024-06-21T01:51:16.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:16] ENGINE Serving on http://:::9283 2024-06-21T01:51:16.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:16] ENGINE Bus STARTED 2024-06-21T01:51:17.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[25107]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-21T01:51:17.681 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:17.681 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:17.681 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:17.681 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:51:17.681 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:17.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[29810]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-21T01:51:17.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:17.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:17.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:17.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:51:17.682 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:17 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:17.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:17 smithi173 ceph-mon[25952]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-21T01:51:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:17 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:17 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:17 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:17 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:51:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:17 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:18.697 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:18 smithi092 ceph-mon[25107]: Upgrade: Updating mgr.y 2024-06-21T01:51:18.697 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:18 smithi092 ceph-mon[25107]: Deploying daemon mgr.y on smithi092 2024-06-21T01:51:18.698 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:18 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1002537221' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:51:18.698 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:18 smithi092 ceph-mon[29810]: Upgrade: Updating mgr.y 2024-06-21T01:51:18.698 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:18 smithi092 ceph-mon[29810]: Deploying daemon mgr.y on smithi092 2024-06-21T01:51:18.698 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:18 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1002537221' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:51:18.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:18 smithi173 ceph-mon[25952]: Upgrade: Updating mgr.y 2024-06-21T01:51:18.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:18 smithi173 ceph-mon[25952]: Deploying daemon mgr.y on smithi092 2024-06-21T01:51:18.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:18 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1002537221' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:51:19.249 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:18 smithi092 systemd[1]: Stopping Ceph mgr.y for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:19.500 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:19 smithi092 podman[59801]: 2024-06-21 01:51:19.253187355 +0000 UTC m=+0.138187130 container died 0b61fab2e6e278ec4c04732ad076bc96f91e693541e72f31beae689d31e4ddde (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.buildah.version=1.19.8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.component=centos-stream-container, GIT_CLEAN=True, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, ceph=True, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-06-21T01:51:19.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:19 smithi092 ceph-mon[29810]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:19.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:19 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1567626000' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3550345633"}]: dispatch 2024-06-21T01:51:19.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:19 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3550345633"}]: dispatch 2024-06-21T01:51:19.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:19 smithi092 ceph-mon[25107]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:19.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:19 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1567626000' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3550345633"}]: dispatch 2024-06-21T01:51:19.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:19 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3550345633"}]: dispatch 2024-06-21T01:51:19.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:19 smithi092 podman[59801]: 2024-06-21 01:51:19.688572885 +0000 UTC m=+0.573572658 container cleanup 0b61fab2e6e278ec4c04732ad076bc96f91e693541e72f31beae689d31e4ddde (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.openshift.tags=base centos centos-stream, distribution-scope=public, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, release=754, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-06-21T01:51:19.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:19 smithi092 bash[59801]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y 2024-06-21T01:51:19.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:19 smithi173 ceph-mon[25952]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:19.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:19 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1567626000' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3550345633"}]: dispatch 2024-06-21T01:51:19.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:19 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3550345633"}]: dispatch 2024-06-21T01:51:20.301 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:19 smithi092 podman[59816]: 2024-06-21 01:51:19.890334481 +0000 UTC m=+0.635841982 container remove 0b61fab2e6e278ec4c04732ad076bc96f91e693541e72f31beae689d31e4ddde (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y, io.openshift.tags=base centos centos-stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , architecture=x86_64, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git) 2024-06-21T01:51:20.301 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:19 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-06-21T01:51:20.301 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-06-21T01:51:20.301 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 systemd[1]: Stopped Ceph mgr.y for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:20.301 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.y.service: Consumed 51.053s CPU time. 2024-06-21T01:51:20.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 systemd[1]: Starting Ceph mgr.y for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:20.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 podman[59963]: 2024-06-21 01:51:20.499489881 +0000 UTC m=+0.093284888 container create fd1d3ac2265b5f453f4a961778918562145111f4be36955f249ff5ccbcfa4fe4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, GIT_CLEAN=True, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-06-21T01:51:20.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 podman[59963]: 2024-06-21 01:51:20.423151089 +0000 UTC m=+0.016946065 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:51:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3550345633"}]': finished 2024-06-21T01:51:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[25107]: osdmap e98: 8 total, 8 up, 8 in 2024-06-21T01:51:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3778775575' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3976715161"}]: dispatch 2024-06-21T01:51:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3976715161"}]: dispatch 2024-06-21T01:51:20.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[25107]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-06-21T01:51:20.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3550345633"}]': finished 2024-06-21T01:51:20.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[29810]: osdmap e98: 8 total, 8 up, 8 in 2024-06-21T01:51:20.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:20.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3778775575' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3976715161"}]: dispatch 2024-06-21T01:51:20.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3976715161"}]: dispatch 2024-06-21T01:51:20.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mon[29810]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-06-21T01:51:20.864 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:20 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3550345633"}]': finished 2024-06-21T01:51:20.864 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:20 smithi173 ceph-mon[25952]: osdmap e98: 8 total, 8 up, 8 in 2024-06-21T01:51:20.864 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:20 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:20.864 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:20 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3778775575' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3976715161"}]: dispatch 2024-06-21T01:51:20.864 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:20 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3976715161"}]: dispatch 2024-06-21T01:51:20.864 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:20 smithi173 ceph-mon[25952]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-06-21T01:51:20.956 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 podman[59963]: 2024-06-21 01:51:20.72684145 +0000 UTC m=+0.320636427 container init fd1d3ac2265b5f453f4a961778918562145111f4be36955f249ff5ccbcfa4fe4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:51:20.956 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 podman[59963]: 2024-06-21 01:51:20.729780098 +0000 UTC m=+0.323575072 container start fd1d3ac2265b5f453f4a961778918562145111f4be36955f249ff5ccbcfa4fe4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y, RELEASE=squid-release-c65be81, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T01:51:20.956 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-mgr[60004]: -- 172.21.15.92:0/3198959629 <== mon.1 v2:172.21.15.92:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5577924d50e0 con 0x5577924b3400 2024-06-21T01:51:20.956 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:20.855+0000 7f80c61b3040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:51:21.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 bash[59963]: fd1d3ac2265b5f453f4a961778918562145111f4be36955f249ff5ccbcfa4fe4 2024-06-21T01:51:21.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:20 smithi092 systemd[1]: Started Ceph mgr.y for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:21.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:21.030+0000 7f80c61b3040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:51:21.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:21.106+0000 7f80c61b3040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:51:21.795 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:21 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3976715161"}]': finished 2024-06-21T01:51:21.795 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:21 smithi173 ceph-mon[25952]: osdmap e99: 8 total, 8 up, 8 in 2024-06-21T01:51:21.795 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:21 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2450845817' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2887174273"}]: dispatch 2024-06-21T01:51:21.795 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:21 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2887174273"}]: dispatch 2024-06-21T01:51:21.795 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:21 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:21.795 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:21 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:21.795 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:21 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:21.795 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:21 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:21.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:21.381+0000 7f80c61b3040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:51:21.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:21.446+0000 7f80c61b3040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:51:21.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3976715161"}]': finished 2024-06-21T01:51:21.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[25107]: osdmap e99: 8 total, 8 up, 8 in 2024-06-21T01:51:21.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/2450845817' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2887174273"}]: dispatch 2024-06-21T01:51:21.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2887174273"}]: dispatch 2024-06-21T01:51:21.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:21.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:21.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:21.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:21.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3976715161"}]': finished 2024-06-21T01:51:21.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[29810]: osdmap e99: 8 total, 8 up, 8 in 2024-06-21T01:51:21.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2450845817' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2887174273"}]: dispatch 2024-06-21T01:51:21.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2887174273"}]: dispatch 2024-06-21T01:51:21.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:21.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:21.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:21.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:21 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:22.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:22 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2887174273"}]': finished 2024-06-21T01:51:22.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:22 smithi173 ceph-mon[25952]: osdmap e100: 8 total, 8 up, 8 in 2024-06-21T01:51:22.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:22 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3080553699' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2887174273"}]: dispatch 2024-06-21T01:51:22.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:22 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2887174273"}]: dispatch 2024-06-21T01:51:22.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:22 smithi173 ceph-mon[25952]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-06-21T01:51:22.832 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2887174273"}]': finished 2024-06-21T01:51:22.832 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-mon[25107]: osdmap e100: 8 total, 8 up, 8 in 2024-06-21T01:51:22.832 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3080553699' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2887174273"}]: dispatch 2024-06-21T01:51:22.832 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2887174273"}]: dispatch 2024-06-21T01:51:22.832 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-mon[25107]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-06-21T01:51:22.832 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/2887174273"}]': finished 2024-06-21T01:51:22.833 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-mon[29810]: osdmap e100: 8 total, 8 up, 8 in 2024-06-21T01:51:22.833 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3080553699' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2887174273"}]: dispatch 2024-06-21T01:51:22.833 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2887174273"}]: dispatch 2024-06-21T01:51:22.833 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-mon[29810]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-06-21T01:51:22.833 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:22.513+0000 7f80c61b3040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:51:22.833 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:22.647+0000 7f80c61b3040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:51:22.833 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:22.713+0000 7f80c61b3040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:51:23.083 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:22 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:22.950+0000 7f80c61b3040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:51:23.084 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:23.014+0000 7f80c61b3040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:51:23.084 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:23.086+0000 7f80c61b3040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:51:23.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:23.247+0000 7f80c61b3040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:51:23.450 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:23 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:51:23] "GET /metrics HTTP/1.1" 200 38150 "" "Prometheus/2.33.4" 2024-06-21T01:51:23.728 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:23 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:51:23.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:23 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2887174273"}]': finished 2024-06-21T01:51:23.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:23 smithi173 ceph-mon[25952]: osdmap e101: 8 total, 8 up, 8 in 2024-06-21T01:51:23.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:23 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/848684377' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/67584640"}]: dispatch 2024-06-21T01:51:23.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:23 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/67584640"}]: dispatch 2024-06-21T01:51:23.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2887174273"}]': finished 2024-06-21T01:51:23.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-mon[29810]: osdmap e101: 8 total, 8 up, 8 in 2024-06-21T01:51:23.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/848684377' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/67584640"}]: dispatch 2024-06-21T01:51:23.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/67584640"}]: dispatch 2024-06-21T01:51:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/2887174273"}]': finished 2024-06-21T01:51:23.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-mon[25107]: osdmap e101: 8 total, 8 up, 8 in 2024-06-21T01:51:23.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/848684377' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/67584640"}]: dispatch 2024-06-21T01:51:23.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/67584640"}]: dispatch 2024-06-21T01:51:23.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:23.716+0000 7f80c61b3040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:51:24.328 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:24.049+0000 7f80c61b3040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:51:24.329 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:24.126+0000 7f80c61b3040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:51:24.329 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:24.289+0000 7f80c61b3040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-21T01:51:24.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/67584640"}]': finished 2024-06-21T01:51:24.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-mon[29810]: osdmap e102: 8 total, 8 up, 8 in 2024-06-21T01:51:24.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1412059532' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1317278911"}]: dispatch 2024-06-21T01:51:24.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1317278911"}]: dispatch 2024-06-21T01:51:24.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-mon[29810]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:24.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/67584640"}]': finished 2024-06-21T01:51:24.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-mon[25107]: osdmap e102: 8 total, 8 up, 8 in 2024-06-21T01:51:24.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1412059532' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1317278911"}]: dispatch 2024-06-21T01:51:24.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1317278911"}]: dispatch 2024-06-21T01:51:24.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-mon[25107]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:24.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:24 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/67584640"}]': finished 2024-06-21T01:51:24.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:24 smithi173 ceph-mon[25952]: osdmap e102: 8 total, 8 up, 8 in 2024-06-21T01:51:24.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:24 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1412059532' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1317278911"}]: dispatch 2024-06-21T01:51:24.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:24 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1317278911"}]: dispatch 2024-06-21T01:51:24.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:24 smithi173 ceph-mon[25952]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:24.910 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:24.631+0000 7f80c61b3040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:51:24.910 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:24.701+0000 7f80c61b3040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:51:25.225 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:24.912+0000 7f80c61b3040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:51:25.225 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:25.069+0000 7f80c61b3040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:51:25.225 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:25.137+0000 7f80c61b3040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:51:25.225 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:25.226+0000 7f80c61b3040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:51:25.537 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:25.292+0000 7f80c61b3040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:51:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1317278911"}]': finished 2024-06-21T01:51:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[25107]: osdmap e103: 8 total, 8 up, 8 in 2024-06-21T01:51:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/528764785' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/779119699"}]: dispatch 2024-06-21T01:51:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/779119699"}]: dispatch 2024-06-21T01:51:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:25.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1317278911"}]': finished 2024-06-21T01:51:25.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[29810]: osdmap e103: 8 total, 8 up, 8 in 2024-06-21T01:51:25.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/528764785' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/779119699"}]: dispatch 2024-06-21T01:51:25.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/779119699"}]: dispatch 2024-06-21T01:51:25.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:25.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:25 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1317278911"}]': finished 2024-06-21T01:51:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:25 smithi173 ceph-mon[25952]: osdmap e103: 8 total, 8 up, 8 in 2024-06-21T01:51:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:25 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/528764785' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/779119699"}]: dispatch 2024-06-21T01:51:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:25 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/779119699"}]: dispatch 2024-06-21T01:51:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:25 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:25 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.167 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:25 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:25.884+0000 7f80c61b3040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:51:26.167 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:26.062+0000 7f80c61b3040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:51:26.167 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:51:26] ENGINE Bus STARTING 2024-06-21T01:51:26.167 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: CherryPy Checker: 2024-06-21T01:51:26.167 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: The Application mounted at '' has an empty config. 2024-06-21T01:51:26.168 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:26.168 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:51:26] ENGINE Serving on http://:::9283 2024-06-21T01:51:26.168 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:51:26] ENGINE Bus STARTED 2024-06-21T01:51:26.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/779119699"}]': finished 2024-06-21T01:51:26.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: osdmap e104: 8 total, 8 up, 8 in 2024-06-21T01:51:26.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: Standby manager daemon y restarted 2024-06-21T01:51:26.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: Standby manager daemon y started 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[25107]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/779119699"}]': finished 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: osdmap e104: 8 total, 8 up, 8 in 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: Standby manager daemon y restarted 2024-06-21T01:51:26.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: Standby manager daemon y started 2024-06-21T01:51:26.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-21T01:51:26.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:51:26.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-21T01:51:26.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:51:26.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:26 smithi092 ceph-mon[29810]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:51:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/779119699"}]': finished 2024-06-21T01:51:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: osdmap e104: 8 total, 8 up, 8 in 2024-06-21T01:51:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: Standby manager daemon y restarted 2024-06-21T01:51:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: Standby manager daemon y started 2024-06-21T01:51:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-21T01:51:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:51:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-21T01:51:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:51:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:26.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:26 smithi173 ceph-mon[25952]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:51:27.813 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:27 smithi092 ceph-mon[25107]: mgrmap e27: x(active, since 53s), standbys: y 2024-06-21T01:51:27.814 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:27 smithi092 ceph-mon[29810]: mgrmap e27: x(active, since 53s), standbys: y 2024-06-21T01:51:27.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:27 smithi173 ceph-mon[25952]: mgrmap e27: x(active, since 53s), standbys: y 2024-06-21T01:51:28.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:28 smithi092 ceph-mon[25107]: from='client.25090 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:51:28.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:28 smithi092 ceph-mon[25107]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-06-21T01:51:28.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:28 smithi092 ceph-mon[29810]: from='client.25090 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:51:28.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:28 smithi092 ceph-mon[29810]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-06-21T01:51:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:28 smithi173 ceph-mon[25952]: from='client.25090 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:51:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:28 smithi173 ceph-mon[25952]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-06-21T01:51:28.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:28 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:51:30.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:30.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:30.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:30.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:51:30.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:30.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[25107]: Reconfiguring iscsi.foo.smithi092.pqnhsu (dependencies changed)... 2024-06-21T01:51:30.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:30.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:30.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:30.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[25107]: Reconfiguring daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:51:30.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:30.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:30.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:30.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:51:30.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:30.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[29810]: Reconfiguring iscsi.foo.smithi092.pqnhsu (dependencies changed)... 2024-06-21T01:51:30.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:30.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:30.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:30.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:30 smithi092 ceph-mon[29810]: Reconfiguring daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:51:30.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:30 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:30.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:30 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:30.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:30 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:30.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:30 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:51:30.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:30 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' 2024-06-21T01:51:30.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:30 smithi173 ceph-mon[25952]: Reconfiguring iscsi.foo.smithi092.pqnhsu (dependencies changed)... 2024-06-21T01:51:30.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:30 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:30.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:30 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:51:30.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:30 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:30.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:30 smithi173 ceph-mon[25952]: Reconfiguring daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:51:31.558 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:31 smithi092 ceph-mon[25107]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-06-21T01:51:31.559 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:31 smithi092 ceph-mon[29810]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-06-21T01:51:31.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:31 smithi173 ceph-mon[25952]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-06-21T01:51:32.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:32.293+0000 7fbc1160f640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T01:51:32.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Creating ceph-iscsi config... 2024-06-21T01:51:32.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T01:51:32.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T01:51:32.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Non-zero exit code 1 from systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T01:51:32.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:32.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:32.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Traceback (most recent call last): 2024-06-21T01:51:32.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T01:51:32.479 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return _run_code(code, main_globals, None, 2024-06-21T01:51:32.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T01:51:32.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: exec(code, run_globals) 2024-06-21T01:51:32.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T01:51:32.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T01:51:32.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T01:51:32.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T01:51:32.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T01:51:32.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1123, in deploy_daemon 2024-06-21T01:51:32.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T01:51:32.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: RuntimeError: Failed command: systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:32.480 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:32.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Traceback (most recent call last): 2024-06-21T01:51:32.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-06-21T01:51:32.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: self.mgr._daemon_action(daemon_spec, action=action) 2024-06-21T01:51:32.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2456, in _daemon_action 2024-06-21T01:51:32.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return self.wait_async( 2024-06-21T01:51:32.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-06-21T01:51:32.481 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return self.event_loop.get_result(coro, timeout) 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return future.result(timeout) 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return self.__get_result() 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: raise self._exception 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1350, in _create_daemon 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: out, err, code = await self._run_cephadm( 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1691, in _run_cephadm 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: raise OrchestratorError( 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Creating ceph-iscsi config... 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T01:51:32.482 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Non-zero exit code 1 from systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: Traceback (most recent call last): 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: return _run_code(code, main_globals, None, 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: exec(code, run_globals) 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T01:51:32.483 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1123, in deploy_daemon 2024-06-21T01:51:32.484 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T01:51:32.484 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: RuntimeError: Failed command: systemctl restart ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:51:32.484 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:51:32.507 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:51:32.507 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-mon[25107]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-21T01:51:32.508 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-21T01:51:32.508 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-mon[25107]: osdmap e105: 8 total, 8 up, 8 in 2024-06-21T01:51:32.508 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:51:32.508 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-mon[29810]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-21T01:51:32.508 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-21T01:51:32.508 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-mon[29810]: osdmap e105: 8 total, 8 up, 8 in 2024-06-21T01:51:32.803 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:51:32.803 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-mon[25952]: from='mgr.24806 172.21.15.173:0/2775577703' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-21T01:51:32.804 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-21T01:51:32.804 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-mon[25952]: osdmap e105: 8 total, 8 up, 8 in 2024-06-21T01:51:32.804 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:32.502+0000 7fbc7c042640 -1 mgr handle_mgr_map I was active but no longer am 2024-06-21T01:51:32.804 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: ignoring --setuser ceph since I am not root 2024-06-21T01:51:32.804 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: ignoring --setgroup ceph since I am not root 2024-06-21T01:51:32.804 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-mgr[47018]: -- 172.21.15.173:0/383458251 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc655030e0 con 0x55dc654e1400 2024-06-21T01:51:32.804 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:32.655+0000 7fe901fe3040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:51:32.804 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:32.806+0000 7fe901fe3040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:51:32.822 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:51:32] ENGINE Bus STOPPING 2024-06-21T01:51:32.822 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:51:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-21T01:51:32.822 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:51:32] ENGINE Bus STOPPED 2024-06-21T01:51:32.822 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:51:32] ENGINE Bus STARTING 2024-06-21T01:51:33.077 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:51:32] ENGINE Serving on http://:::9283 2024-06-21T01:51:33.077 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:32 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:51:32] ENGINE Bus STARTED 2024-06-21T01:51:33.123 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:32 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:32.876+0000 7fe901fe3040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:51:33.123 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:33.126+0000 7fe901fe3040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:51:33.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:33.190+0000 7fe901fe3040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:51:33.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.24806 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: mgrmap e28: y(active, starting, since 0.168474s) 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:51:33.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: Manager daemon y is now available 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.24806 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: mgrmap e28: y(active, starting, since 0.168474s) 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:51:33.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:51:33.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:51:33.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:51:33.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: Manager daemon y is now available 2024-06-21T01:51:33.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:33.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:51:33.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:51:33.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:51:33.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:33 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:51:33.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.24806 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: mgrmap e28: y(active, starting, since 0.168474s) 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:51:33.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:51:33.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:51:33.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:51:33.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:51:33.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:51:33.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: Manager daemon y is now available 2024-06-21T01:51:33.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:33.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:51:33.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:51:33.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:51:33.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:33 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:51:34.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:34.016+0000 7f801dca3640 -1 log_channel(cephadm) log [ERR] : [21/Jun/2024:01:51:34] ENGINE Error in HTTPServer.serve 2024-06-21T01:51:34.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: Traceback (most recent call last): 2024-06-21T01:51:34.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:51:34.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: self._connections.run(self.expiration_interval) 2024-06-21T01:51:34.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:51:34.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: self._run(expiration_interval) 2024-06-21T01:51:34.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:51:34.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:51:34.353 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:51:34.353 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:51:34.353 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:51:34.353 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: s = self.context.wrap_socket( 2024-06-21T01:51:34.353 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:51:34.353 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: return self.sslsocket_class._create( 2024-06-21T01:51:34.353 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:51:34.353 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: self.do_handshake() 2024-06-21T01:51:34.353 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:51:34.353 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: self._sslobj.do_handshake() 2024-06-21T01:51:34.353 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:51:34.354 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:51:34.399 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:34.217+0000 7fe901fe3040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:51:34.399 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:34.342+0000 7fe901fe3040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:51:34.399 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:34.402+0000 7fe901fe3040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:51:34.649 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:34 smithi173 ceph-mon[25952]: mgrmap e29: y(active, since 1.17389s) 2024-06-21T01:51:34.649 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:34 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:34.650 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:34 smithi173 ceph-mon[25952]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:34.650 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:34 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:34.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-mon[25107]: mgrmap e29: y(active, since 1.17389s) 2024-06-21T01:51:34.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:34.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-mon[25107]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:34.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:34.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-mon[29810]: mgrmap e29: y(active, since 1.17389s) 2024-06-21T01:51:34.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:34.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-mon[29810]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:34.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:34 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:34.915 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:34.652+0000 7fe901fe3040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:51:34.915 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:34.716+0000 7fe901fe3040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:51:34.915 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:34.784+0000 7fe901fe3040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:51:35.192 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:34.942+0000 7fe901fe3040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:51:35.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:35 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:35.377+0000 7fe901fe3040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:51:35.726 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:51:35.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[25107]: [21/Jun/2024:01:51:33] ENGINE Bus STARTING 2024-06-21T01:51:35.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[25107]: [21/Jun/2024:01:51:33] ENGINE Serving on http://172.21.15.92:8765 2024-06-21T01:51:35.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[25107]: [21/Jun/2024:01:51:34] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:51:35.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[25107]: [21/Jun/2024:01:51:34] ENGINE Bus STARTED 2024-06-21T01:51:35.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[25107]: [21/Jun/2024:01:51:34] ENGINE Error in HTTPServer.serve 2024-06-21T01:51:35.727 INFO:journalctl@ceph.mon.a.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:51:35.727 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:51:35.727 INFO:journalctl@ceph.mon.a.smithi092.stdout: self._connections.run(self.expiration_interval) 2024-06-21T01:51:35.727 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:51:35.727 INFO:journalctl@ceph.mon.a.smithi092.stdout: self._run(expiration_interval) 2024-06-21T01:51:35.727 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:51:35.727 INFO:journalctl@ceph.mon.a.smithi092.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:51:35.727 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:51:35.728 INFO:journalctl@ceph.mon.a.smithi092.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:51:35.728 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:51:35.728 INFO:journalctl@ceph.mon.a.smithi092.stdout: s = self.context.wrap_socket( 2024-06-21T01:51:35.728 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:51:35.728 INFO:journalctl@ceph.mon.a.smithi092.stdout: return self.sslsocket_class._create( 2024-06-21T01:51:35.728 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:51:35.728 INFO:journalctl@ceph.mon.a.smithi092.stdout: self.do_handshake() 2024-06-21T01:51:35.728 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:51:35.728 INFO:journalctl@ceph.mon.a.smithi092.stdout: self._sslobj.do_handshake() 2024-06-21T01:51:35.728 INFO:journalctl@ceph.mon.a.smithi092.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:51:35.728 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[25107]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:35.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[25107]: mgrmap e30: y(active, since 2s) 2024-06-21T01:51:35.729 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[29810]: [21/Jun/2024:01:51:33] ENGINE Bus STARTING 2024-06-21T01:51:35.729 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[29810]: [21/Jun/2024:01:51:33] ENGINE Serving on http://172.21.15.92:8765 2024-06-21T01:51:35.729 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[29810]: [21/Jun/2024:01:51:34] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:51:35.729 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[29810]: [21/Jun/2024:01:51:34] ENGINE Bus STARTED 2024-06-21T01:51:35.729 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[29810]: [21/Jun/2024:01:51:34] ENGINE Error in HTTPServer.serve 2024-06-21T01:51:35.729 INFO:journalctl@ceph.mon.c.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:51:35.729 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:51:35.729 INFO:journalctl@ceph.mon.c.smithi092.stdout: self._connections.run(self.expiration_interval) 2024-06-21T01:51:35.729 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:51:35.730 INFO:journalctl@ceph.mon.c.smithi092.stdout: self._run(expiration_interval) 2024-06-21T01:51:35.730 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:51:35.730 INFO:journalctl@ceph.mon.c.smithi092.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:51:35.730 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:51:35.730 INFO:journalctl@ceph.mon.c.smithi092.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:51:35.730 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:51:35.730 INFO:journalctl@ceph.mon.c.smithi092.stdout: s = self.context.wrap_socket( 2024-06-21T01:51:35.730 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:51:35.730 INFO:journalctl@ceph.mon.c.smithi092.stdout: return self.sslsocket_class._create( 2024-06-21T01:51:35.730 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:51:35.730 INFO:journalctl@ceph.mon.c.smithi092.stdout: self.do_handshake() 2024-06-21T01:51:35.731 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:51:35.731 INFO:journalctl@ceph.mon.c.smithi092.stdout: self._sslobj.do_handshake() 2024-06-21T01:51:35.731 INFO:journalctl@ceph.mon.c.smithi092.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:51:35.731 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[29810]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:35.731 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:35 smithi092 ceph-mon[29810]: mgrmap e30: y(active, since 2s) 2024-06-21T01:51:35.822 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:35 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:35.757+0000 7fe901fe3040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:51:35.822 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:35 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:35.825+0000 7fe901fe3040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:35 smithi173 ceph-mon[25952]: [21/Jun/2024:01:51:33] ENGINE Bus STARTING 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:35 smithi173 ceph-mon[25952]: [21/Jun/2024:01:51:33] ENGINE Serving on http://172.21.15.92:8765 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:35 smithi173 ceph-mon[25952]: [21/Jun/2024:01:51:34] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:35 smithi173 ceph-mon[25952]: [21/Jun/2024:01:51:34] ENGINE Bus STARTED 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:35 smithi173 ceph-mon[25952]: [21/Jun/2024:01:51:34] ENGINE Error in HTTPServer.serve 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: Traceback (most recent call last): 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._connections.run(self.expiration_interval) 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._run(expiration_interval) 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:51:35.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: s = self.context.wrap_socket( 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: return self.sslsocket_class._create( 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: self.do_handshake() 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._sslobj.do_handshake() 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:35 smithi173 ceph-mon[25952]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:35.824 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:35 smithi173 ceph-mon[25952]: mgrmap e30: y(active, since 2s) 2024-06-21T01:51:36.086 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:35 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:35.991+0000 7fe901fe3040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-21T01:51:36.385 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:36.322+0000 7fe901fe3040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:51:36.391 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (35s) 10s ago 4m 14.7M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (21s) 11s ago 4m 41.2M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (19s) 10s ago 4m 58.8M - 3.5 e1d6a67b021e f4141b8de1af 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283 running (71s) 11s ago 9m 504M - 19.0.0-2541-gc65be810 e9bfc9e474e5 d64fa854065a 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (15s) 10s ago 11m 92.2M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (11m) 10s ago 11m 54.0M 2048M 17.2.0 e1d6a67b021e 3daa4a891e57 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (9m) 11s ago 9m 46.7M 2048M 17.2.0 e1d6a67b021e 00fcf1f13930 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (9m) 10s ago 9m 43.4M 2048M 17.2.0 e1d6a67b021e 6a24d6b0bbb6 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (39s) 10s ago 5m 8526k - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (25s) 11s ago 5m 9265k - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (9m) 10s ago 9m 52.4M 2736M 17.2.0 e1d6a67b021e 63507222842b 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (8m) 10s ago 8m 54.3M 2736M 17.2.0 e1d6a67b021e e0bf7c0600e5 2024-06-21T01:51:36.392 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (8m) 10s ago 8m 47.3M 2736M 17.2.0 e1d6a67b021e 157fb16866c9 2024-06-21T01:51:36.393 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (8m) 10s ago 8m 50.7M 2736M 17.2.0 e1d6a67b021e e5567e44eaec 2024-06-21T01:51:36.393 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (7m) 11s ago 7m 54.4M 2992M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:51:36.393 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (7m) 11s ago 7m 52.5M 2992M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:51:36.393 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (6m) 11s ago 6m 48.8M 2992M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:51:36.393 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (6m) 11s ago 6m 51.6M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:51:36.393 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (27s) 11s ago 5m 46.7M - 2.33.4 514e6a882f6e 77bcbddbb5ac 2024-06-21T01:51:36.393 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (4m) 10s ago 4m 74.5M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:51:36.393 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (4m) 11s ago 4m 71.1M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:51:36.393 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (4m) 10s ago 4m 71.9M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:51:36.393 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (4m) 11s ago 4m 72.5M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:51:36.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:36 smithi092 ceph-mon[25107]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:36.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:36 smithi092 ceph-mon[29810]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:36.682 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:36 smithi173 ceph-mon[25952]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:36.682 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:36.388+0000 7fe901fe3040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:51:36.682 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:36.584+0000 7fe901fe3040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:51:36.844 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:51:36.844 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:51:36.844 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-06-21T01:51:36.844 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:51:36.844 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 1 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": {}, 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 1 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:51:36.845 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:51:36.970 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:36.738+0000 7fe901fe3040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:51:36.970 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:36.804+0000 7fe901fe3040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:51:36.970 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:36.889+0000 7fe901fe3040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:51:36.970 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:36.972+0000 7fe901fe3040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:51:37.212 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:51:37.212 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:51:37.212 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:51:37.212 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:51:37.212 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:51:37.212 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr" 2024-06-21T01:51:37.213 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:51:37.213 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "2/23 daemons upgraded", 2024-06-21T01:51:37.213 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "", 2024-06-21T01:51:37.213 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:51:37.213 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:51:37.648 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:51:37.835 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-mon[25952]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:37.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-mon[25952]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:37.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-mon[25952]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:37.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-mon[25952]: mgrmap e31: y(active, since 4s) 2024-06-21T01:51:37.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3088295753' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:51:37.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:37.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:37.836 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:37.559+0000 7fe901fe3040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:51:37.836 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:37.730+0000 7fe901fe3040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:51:37.836 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:37] ENGINE Bus STARTING 2024-06-21T01:51:37.836 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: CherryPy Checker: 2024-06-21T01:51:37.836 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: The Application mounted at '' has an empty config. 2024-06-21T01:51:37.837 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: 2024-06-21T01:51:37.837 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:37] ENGINE Serving on http://:::9283 2024-06-21T01:51:37.837 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:37 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: [21/Jun/2024:01:51:37] ENGINE Bus STARTED 2024-06-21T01:51:37.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[29810]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:37.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[29810]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:37.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[29810]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:37.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[29810]: mgrmap e31: y(active, since 4s) 2024-06-21T01:51:37.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3088295753' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:51:37.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:37.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:37.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[25107]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:37.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[25107]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:37.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[25107]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:37.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[25107]: mgrmap e31: y(active, since 4s) 2024-06-21T01:51:37.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/3088295753' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:51:37.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:37.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:37 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/1262060391' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:51:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: Standby manager daemon x started 2024-06-21T01:51:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:51:38.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:51:38.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:51:38.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:51:38.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:38.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/1262060391' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:51:38.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: Standby manager daemon x started 2024-06-21T01:51:38.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:51:38.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:51:38.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:51:38.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:51:38.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:38 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:51:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/1262060391' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:51:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: Standby manager daemon x started 2024-06-21T01:51:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:51:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:51:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:51:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/3782936313' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:51:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:38.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:38 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:39.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:39 smithi092 ceph-mon[29810]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:39.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:39 smithi092 ceph-mon[29810]: mgrmap e32: y(active, since 6s), standbys: x 2024-06-21T01:51:39.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:39 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:51:39.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:39 smithi092 ceph-mon[25107]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:39.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:39 smithi092 ceph-mon[25107]: mgrmap e32: y(active, since 6s), standbys: x 2024-06-21T01:51:39.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:39 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:51:39.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:39 smithi173 ceph-mon[25952]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:51:39.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:39 smithi173 ceph-mon[25952]: mgrmap e32: y(active, since 6s), standbys: x 2024-06-21T01:51:39.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:39 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:51:41.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:41 smithi092 ceph-mon[29810]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-06-21T01:51:41.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:41 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:41.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:41 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:41.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:41 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:41.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:41 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:51:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:41 smithi092 ceph-mon[25107]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-06-21T01:51:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:41 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:41 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:41 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:41.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:41 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:51:41.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:41 smithi173 ceph-mon[25952]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-06-21T01:51:41.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:41 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:41.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:41 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:41.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:41 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:41.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:41 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:51:42.699 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:51:42.699 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:51:42.699 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:51:42.699 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:51:42.699 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:51:42.699 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:51:42.699 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:51:42.699 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:51:42.699 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.699 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.700 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.700 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.700 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:42 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:51:42.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:51:42.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:51:42.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:51:42.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:51:42.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:51:42.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:51:42.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:51:42.710 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:51:42.710 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.710 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.710 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.710 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:42.710 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:42 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:43.472 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:43 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[47014]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:51:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-21T01:51:43.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:43 smithi173 ceph-mon[25952]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-06-21T01:51:43.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:43 smithi173 ceph-mon[25952]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-21T01:51:43.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:43 smithi173 ceph-mon[25952]: Reconfiguring daemon prometheus.a on smithi173 2024-06-21T01:51:43.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:43 smithi173 ceph-mon[25952]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-21T01:51:43.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:43 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:51:43.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:43 smithi092 ceph-mon[25107]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-06-21T01:51:43.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:43 smithi092 ceph-mon[25107]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-21T01:51:43.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:43 smithi092 ceph-mon[25107]: Reconfiguring daemon prometheus.a on smithi173 2024-06-21T01:51:43.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:43 smithi092 ceph-mon[25107]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-21T01:51:43.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:43 smithi092 ceph-mon[29810]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-06-21T01:51:43.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:43 smithi092 ceph-mon[29810]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-21T01:51:43.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:43 smithi092 ceph-mon[29810]: Reconfiguring daemon prometheus.a on smithi173 2024-06-21T01:51:43.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:43 smithi092 ceph-mon[29810]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-21T01:51:44.780 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:44 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/198968452' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:51:44.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:44 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/198968452' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:51:44.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:44 smithi092 ceph-mon[25107]: from='client.? 172.21.15.92:0/198968452' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:51:45.043 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:44 smithi173 systemd[1]: Stopping Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:45.043 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:44 smithi173 bash[53481]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus.a" found: no such container 2024-06-21T01:51:45.385 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.046Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-21T01:51:45.385 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.046Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-06-21T01:51:45.385 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.046Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-06-21T01:51:45.385 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.046Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-06-21T01:51:45.385 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.046Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-06-21T01:51:45.385 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.046Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-06-21T01:51:45.385 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.046Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-21T01:51:45.385 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.046Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-06-21T01:51:45.386 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.046Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-06-21T01:51:45.386 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.047Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-06-21T01:51:45.386 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.047Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-06-21T01:51:45.386 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[50397]: ts=2024-06-21T01:51:45.047Z caller=main.go:1066 level=info msg="See you next time!" 2024-06-21T01:51:45.386 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 podman[53492]: 2024-06-21 01:51:45.056894317 +0000 UTC m=+0.110307870 container died 77bcbddbb5ac896c05a81a67a669be6cf3a5ad0caed7bc9502954d84a323c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:45.677 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:45 smithi173 ceph-mon[25952]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-21T01:51:45.677 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 podman[53492]: 2024-06-21 01:51:45.47951433 +0000 UTC m=+0.532927882 container cleanup 77bcbddbb5ac896c05a81a67a669be6cf3a5ad0caed7bc9502954d84a323c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:45.677 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 bash[53492]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a 2024-06-21T01:51:45.677 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 bash[53518]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus.a" found: no such container 2024-06-21T01:51:45.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:45 smithi092 ceph-mon[29810]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-21T01:51:45.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:45 smithi092 ceph-mon[25107]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-21T01:51:45.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 podman[53506]: 2024-06-21 01:51:45.680963607 +0000 UTC m=+0.620918973 container remove 77bcbddbb5ac896c05a81a67a669be6cf3a5ad0caed7bc9502954d84a323c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:45.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-06-21T01:51:45.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 systemd[1]: Stopped Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:45.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 systemd[1]: Starting Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:45.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 podman[53581]: 2024-06-21 01:51:45.915231085 +0000 UTC m=+0.114097116 container create 8e7ba220b26dccc5f14aa0d5d9628e72630745d2d847e428b52499c3bc4b0a0f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:45.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:45 smithi173 podman[53581]: 2024-06-21 01:51:45.81988457 +0000 UTC m=+0.018750560 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-06-21T01:51:46.290 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 podman[53581]: 2024-06-21 01:51:46.267590416 +0000 UTC m=+0.466456416 container init 8e7ba220b26dccc5f14aa0d5d9628e72630745d2d847e428b52499c3bc4b0a0f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:46.290 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 podman[53581]: 2024-06-21 01:51:46.270411392 +0000 UTC m=+0.469277381 container start 8e7ba220b26dccc5f14aa0d5d9628e72630745d2d847e428b52499c3bc4b0a0f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.294Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.294Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.294Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.294Z caller=main.go:518 level=info host_details="(Linux 5.14.0-460.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024 x86_64 smithi173 (none))" 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.294Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.294Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.296Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.296Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.297Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.299Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.299Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.604µs 2024-06-21T01:51:46.729 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.299Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-06-21T01:51:46.730 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 bash[53581]: 8e7ba220b26dccc5f14aa0d5d9628e72630745d2d847e428b52499c3bc4b0a0f 2024-06-21T01:51:46.730 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 systemd[1]: Started Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:46.730 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.525Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-06-21T01:51:47.009 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.760Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-06-21T01:51:47.009 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.973Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-06-21T01:51:47.009 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.973Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-06-21T01:51:47.009 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.973Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=37.696µs wal_replay_duration=673.829786ms total_replay_duration=673.88533ms 2024-06-21T01:51:47.010 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.978Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-06-21T01:51:47.010 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.978Z caller=main.go:947 level=info msg="TSDB started" 2024-06-21T01:51:47.010 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.978Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-06-21T01:51:47.010 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.997Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.245397ms db_storage=819ns remote_storage=1.295µs web_handler=593ns query_engine=668ns scrape=403.849µs scrape_sd=124.32µs notify=34.161µs notify_sd=17.765µs rules=18.244034ms 2024-06-21T01:51:47.010 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:46.997Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-06-21T01:51:47.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:47 smithi173 ceph-mon[25952]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:51:47.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:47 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:47.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:47 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:47.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:47 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:47.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:47 smithi173 ceph-mon[25952]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:47.638 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:47 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:51:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[25107]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:51:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[25107]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:47.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:51:47.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[29810]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:51:47.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:47.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:47.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:47.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[29810]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-21T01:51:47.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:47 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:51:48.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:48 smithi173 ceph-mon[25952]: Upgrade: Updating mgr.x 2024-06-21T01:51:48.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:48 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:48.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:48 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:48.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:48 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:51:48.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:48 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:48.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:48 smithi173 ceph-mon[25952]: Deploying daemon mgr.x on smithi173 2024-06-21T01:51:48.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:48 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:48.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:48 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:48.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[25107]: Upgrade: Updating mgr.x 2024-06-21T01:51:48.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:48.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:48.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:51:48.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:48.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[25107]: Deploying daemon mgr.x on smithi173 2024-06-21T01:51:48.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:48.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:48.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[29810]: Upgrade: Updating mgr.x 2024-06-21T01:51:48.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:48.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:51:48.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:51:48.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:51:48.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[29810]: Deploying daemon mgr.x on smithi173 2024-06-21T01:51:48.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:48.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:48 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:51:49.447 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:49 smithi173 ceph-mon[25952]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:51:49.447 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:49 smithi173 systemd[1]: Stopping Ceph mgr.x for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:49.714 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:49 smithi173 podman[53930]: 2024-06-21 01:51:49.450178807 +0000 UTC m=+0.145560147 container died d64fa854065ae1386f499bf5474120a0892abb0e833eed0e54a61eea232cd091 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-21T01:51:49.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:49 smithi092 ceph-mon[25107]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:51:49.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:49 smithi092 ceph-mon[29810]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:51:49.967 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:49 smithi173 podman[53930]: 2024-06-21 01:51:49.718706629 +0000 UTC m=+0.414087965 container cleanup d64fa854065ae1386f499bf5474120a0892abb0e833eed0e54a61eea232cd091 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-06-21T01:51:49.967 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:49 smithi173 bash[53930]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x 2024-06-21T01:51:50.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:49 smithi173 podman[53944]: 2024-06-21 01:51:49.970432938 +0000 UTC m=+0.519995206 container remove d64fa854065ae1386f499bf5474120a0892abb0e833eed0e54a61eea232cd091 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:51:50.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:49 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-06-21T01:51:50.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-06-21T01:51:50.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 systemd[1]: Stopped Ceph mgr.x for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:50.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.x.service: Consumed 18.015s CPU time. 2024-06-21T01:51:50.656 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 systemd[1]: Starting Ceph mgr.x for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:51:50.657 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 podman[54070]: 2024-06-21 01:51:50.45812084 +0000 UTC m=+0.017513943 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:51:50.657 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 podman[54070]: 2024-06-21 01:51:50.563045809 +0000 UTC m=+0.122438899 container create 6e42ae7481a01d19551742a8b82fbd5a97ef32646983bd6036cfb2343d6cddff (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240618, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-06-21T01:51:51.080 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 podman[54070]: 2024-06-21 01:51:50.79049885 +0000 UTC m=+0.349891953 container init 6e42ae7481a01d19551742a8b82fbd5a97ef32646983bd6036cfb2343d6cddff (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-21T01:51:51.080 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 podman[54070]: 2024-06-21 01:51:50.793397329 +0000 UTC m=+0.352790428 container start 6e42ae7481a01d19551742a8b82fbd5a97ef32646983bd6036cfb2343d6cddff (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, ceph=True, GIT_BRANCH=HEAD) 2024-06-21T01:51:51.080 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 ceph-mgr[54089]: -- 172.21.15.173:0/1785143695 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c04a61b0e0 con 0x55c04a5f9400 2024-06-21T01:51:51.080 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 bash[54070]: 6e42ae7481a01d19551742a8b82fbd5a97ef32646983bd6036cfb2343d6cddff 2024-06-21T01:51:51.080 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 systemd[1]: Started Ceph mgr.x for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:51:51.080 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:50 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:50.922+0000 7fe3cedff040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:51:51.080 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:51 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:51.082+0000 7fe3cedff040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:51:51.364 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:51 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:51.160+0000 7fe3cedff040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:51:51.639 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:51 smithi173 ceph-mon[25952]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:51:51.639 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:51 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:51.639 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:51 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:51.640 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:51 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:51.441+0000 7fe3cedff040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:51:51.640 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:51 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:51.504+0000 7fe3cedff040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:51:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:51 smithi092 ceph-mon[25107]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:51:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:51 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:51 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:51.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:51 smithi092 ceph-mon[29810]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:51:51.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:51 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:51.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:51 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:52.638 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:52.520+0000 7fe3cedff040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:51:52.639 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:52.641+0000 7fe3cedff040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:51:52.946 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:52.703+0000 7fe3cedff040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:51:52.946 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:52.948+0000 7fe3cedff040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:51:53.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:53.020+0000 7fe3cedff040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:51:53.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:53.096+0000 7fe3cedff040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:51:53.500 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:53 smithi173 ceph-mon[25952]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 0 op/s 2024-06-21T01:51:53.501 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:53.252+0000 7fe3cedff040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:51:53.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:53 smithi092 ceph-mon[25107]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 0 op/s 2024-06-21T01:51:53.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:53 smithi092 ceph-mon[29810]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 0 op/s 2024-06-21T01:51:53.918 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:51:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:51:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:51:53.918 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:53.695+0000 7fe3cedff040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:51:54.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:54 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:54.033+0000 7fe3cedff040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:51:54.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:54 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:54.102+0000 7fe3cedff040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:51:54.591 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:54 smithi173 ceph-mon[25952]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:51:54.591 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:54 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:54.259+0000 7fe3cedff040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-21T01:51:54.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:54 smithi092 ceph-mon[25107]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:51:54.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:54 smithi092 ceph-mon[29810]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:51:54.853 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:54 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:54.594+0000 7fe3cedff040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:51:54.853 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:54 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:54.667+0000 7fe3cedff040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:51:55.171 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:54 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:54.857+0000 7fe3cedff040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:51:55.172 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:55 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:55.013+0000 7fe3cedff040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:51:55.172 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:55 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:55.079+0000 7fe3cedff040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:51:55.172 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:55 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:55.174+0000 7fe3cedff040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:51:55.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:55 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:55.241+0000 7fe3cedff040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:51:55.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:55 smithi173 ceph-mon[25952]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:55.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:55 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:55.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:55 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:55.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:55 smithi092 ceph-mon[25107]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:55.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:55 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:55.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:55 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:55.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:55 smithi092 ceph-mon[29810]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:55.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:55 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:55.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:55 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:56.113 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:55 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:55.833+0000 7fe3cedff040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:51:56.114 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:56.010+0000 7fe3cedff040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:51:56.114 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: [21/Jun/2024:01:51:56] ENGINE Bus STARTING 2024-06-21T01:51:56.114 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: CherryPy Checker: 2024-06-21T01:51:56.114 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: The Application mounted at '' has an empty config. 2024-06-21T01:51:56.114 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:51:56.114 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: [21/Jun/2024:01:51:56] ENGINE Serving on http://:::9283 2024-06-21T01:51:56.114 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: [21/Jun/2024:01:51:56] ENGINE Bus STARTED 2024-06-21T01:51:57.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[25107]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:57.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:57.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:57.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[25107]: Standby manager daemon x restarted 2024-06-21T01:51:57.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[25107]: Standby manager daemon x started 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[25107]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[29810]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[29810]: Standby manager daemon x restarted 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[29810]: Standby manager daemon x started 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:51:57.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:51:57.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:56 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:51:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-mon[25952]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:51:57.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-mon[25952]: Standby manager daemon x restarted 2024-06-21T01:51:57.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-mon[25952]: Standby manager daemon x started 2024-06-21T01:51:57.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:51:57.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:51:57.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:51:57.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:56 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/2772992376' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:51:57.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:57 smithi173 ceph-mon[25952]: mgrmap e33: y(active, since 24s), standbys: x 2024-06-21T01:51:58.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:57 smithi092 ceph-mon[25107]: mgrmap e33: y(active, since 24s), standbys: x 2024-06-21T01:51:58.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:57 smithi092 ceph-mon[29810]: mgrmap e33: y(active, since 24s), standbys: x 2024-06-21T01:51:59.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:51:58 smithi092 ceph-mon[25107]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:59.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:51:58 smithi092 ceph-mon[29810]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:59.181 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:51:58 smithi173 ceph-mon[25952]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:51:59.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:51:59 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:51:59] "GET /metrics HTTP/1.1" 200 38234 "" "Prometheus/2.33.4" 2024-06-21T01:52:00.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:00.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:00.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-21T01:52:00.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:52:00.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:52:00.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:00.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: Upgrade: Setting container_image for all mgr 2024-06-21T01:52:00.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-21T01:52:00.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-21T01:52:00.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-06-21T01:52:00.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-21T01:52:00.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-21T01:52:00.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-06-21T01:52:00.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-21T01:52:00.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-06-21T01:52:00.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[25107]: Upgrade: It appears safe to stop mon.a 2024-06-21T01:52:00.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: Upgrade: Setting container_image for all mgr 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-06-21T01:52:00.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-21T01:52:00.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-21T01:52:00.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-06-21T01:52:00.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-21T01:52:00.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-06-21T01:52:00.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:00 smithi092 ceph-mon[29810]: Upgrade: It appears safe to stop mon.a 2024-06-21T01:52:00.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: Upgrade: Setting container_image for all mgr 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-21T01:52:00.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-21T01:52:00.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-06-21T01:52:00.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-21T01:52:00.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-06-21T01:52:00.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:00 smithi173 ceph-mon[25952]: Upgrade: It appears safe to stop mon.a 2024-06-21T01:52:02.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[25107]: Upgrade: Updating mon.a 2024-06-21T01:52:02.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[25107]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:02.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:02.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:02.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[25107]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:02.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[25107]: Deploying daemon mon.a on smithi092 2024-06-21T01:52:02.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[25107]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:52:02.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[29810]: Upgrade: Updating mon.a 2024-06-21T01:52:02.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:02.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:02.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:02.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:02.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[29810]: Deploying daemon mon.a on smithi092 2024-06-21T01:52:02.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:01 smithi092 ceph-mon[29810]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:52:02.133 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:01 smithi173 ceph-mon[25952]: Upgrade: Updating mon.a 2024-06-21T01:52:02.133 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:01 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='mgr.y' 2024-06-21T01:52:02.133 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:01 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:02.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:01 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:02.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:01 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:02.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:01 smithi173 ceph-mon[25952]: Deploying daemon mon.a on smithi092 2024-06-21T01:52:02.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:01 smithi173 ceph-mon[25952]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:52:02.720 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:02 smithi092 systemd[1]: Stopping Ceph mon.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:52:03.077 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:02 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a[25103]: 2024-06-21T01:52:02.721+0000 7f8b79c97700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:52:03.078 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:02 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a[25103]: 2024-06-21T01:52:02.721+0000 7f8b79c97700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-06-21T01:52:03.078 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:02 smithi092 podman[64117]: 2024-06-21 01:52:02.814568592 +0000 UTC m=+0.208475647 container died 3daa4a891e57574c0c016dd9c8883dc298b55abb936cbcf1163ec8973d8ae928 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_BRANCH=HEAD, RELEASE=HEAD, vendor=Red Hat, Inc., ceph=True, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.openshift.expose-services=, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-06-21T01:52:03.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:03 smithi092 podman[64117]: 2024-06-21 01:52:03.264239321 +0000 UTC m=+0.658146374 container cleanup 3daa4a891e57574c0c016dd9c8883dc298b55abb936cbcf1163ec8973d8ae928 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, vendor=Red Hat, Inc., io.buildah.version=1.19.8, ceph=True, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, release=754, com.redhat.component=centos-stream-container, vcs-type=git, RELEASE=HEAD, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-06-21T01:52:03.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:03 smithi092 bash[64117]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a 2024-06-21T01:52:03.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:03 smithi092 podman[64129]: 2024-06-21 01:52:03.532283717 +0000 UTC m=+0.714831322 container remove 3daa4a891e57574c0c016dd9c8883dc298b55abb936cbcf1163ec8973d8ae928 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, distribution-scope=public, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, release=754, com.redhat.component=centos-stream-container, io.openshift.expose-services=, ceph=True, io.openshift.tags=base centos centos-stream, name=centos-stream, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-06-21T01:52:03.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:03 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-21T01:52:03.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:03 smithi092 systemd[1]: Stopped Ceph mon.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:52:03.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:03 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.a.service: Consumed 12.986s CPU time. 2024-06-21T01:52:03.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:52:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:52:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:52:04.253 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:03 smithi092 systemd[1]: Starting Ceph mon.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:52:04.253 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 podman[64257]: 2024-06-21 01:52:04.013065887 +0000 UTC m=+0.017538237 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:52:04.253 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 podman[64257]: 2024-06-21 01:52:04.1418423 +0000 UTC m=+0.146314642 container create 347e7d52844ca86ab53047fe174c28d99da1cfa0d72b21cfbd244b8734bca3b0 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=) 2024-06-21T01:52:04.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 podman[64257]: 2024-06-21 01:52:04.378126264 +0000 UTC m=+0.382598606 container init 347e7d52844ca86ab53047fe174c28d99da1cfa0d72b21cfbd244b8734bca3b0 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, ceph=True) 2024-06-21T01:52:04.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 podman[64257]: 2024-06-21 01:52:04.381895653 +0000 UTC m=+0.386367996 container start 347e7d52844ca86ab53047fe174c28d99da1cfa0d72b21cfbd244b8734bca3b0 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81) 2024-06-21T01:52:04.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: set uid:gid to 167:167 (ceph:ceph) 2024-06-21T01:52:04.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc), process ceph-mon, pid 2 2024-06-21T01:52:04.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: pidfile_write: ignore empty --pid-file 2024-06-21T01:52:04.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: load: jerasure load: lrc 2024-06-21T01:52:04.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: RocksDB version: 7.9.2 2024-06-21T01:52:04.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Git sha 0 2024-06-21T01:52:04.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Compile date 2024-06-19 15:40:36 2024-06-21T01:52:04.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: DB SUMMARY 2024-06-21T01:52:04.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: DB Session ID: 4MVZDT9W1KNG8J5VN0ZZ 2024-06-21T01:52:04.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: CURRENT file: CURRENT 2024-06-21T01:52:04.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: IDENTITY file: IDENTITY 2024-06-21T01:52:04.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: MANIFEST file: MANIFEST-000015 size: 980 Bytes 2024-06-21T01:52:04.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-06-21T01:52:04.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 2678630 ; 2024-06-21T01:52:04.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.error_if_exists: 0 2024-06-21T01:52:04.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.create_if_missing: 0 2024-06-21T01:52:04.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.paranoid_checks: 1 2024-06-21T01:52:04.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-21T01:52:04.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-21T01:52:04.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-21T01:52:04.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.env: 0x55a524c51740 2024-06-21T01:52:04.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.fs: PosixFileSystem 2024-06-21T01:52:04.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.info_log: 0x55a5269a70c0 2024-06-21T01:52:04.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_file_opening_threads: 16 2024-06-21T01:52:04.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.statistics: (nil) 2024-06-21T01:52:04.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.use_fsync: 0 2024-06-21T01:52:04.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_log_file_size: 0 2024-06-21T01:52:04.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-21T01:52:04.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-21T01:52:04.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.keep_log_file_num: 1000 2024-06-21T01:52:04.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.recycle_log_file_num: 0 2024-06-21T01:52:04.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.allow_fallocate: 1 2024-06-21T01:52:04.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.allow_mmap_reads: 0 2024-06-21T01:52:04.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.allow_mmap_writes: 0 2024-06-21T01:52:04.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.use_direct_reads: 0 2024-06-21T01:52:04.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-21T01:52:04.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.create_missing_column_families: 0 2024-06-21T01:52:04.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.db_log_dir: 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.wal_dir: 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.advise_random_on_open: 1 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.db_write_buffer_size: 0 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.write_buffer_manager: 0x55a5269b6c80 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-21T01:52:04.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.rate_limiter: (nil) 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.wal_recovery_mode: 2 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.enable_thread_tracking: 0 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.enable_pipelined_write: 0 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.unordered_write: 0 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.row_cache: None 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.wal_filter: None 2024-06-21T01:52:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-21T01:52:04.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.allow_ingest_behind: 0 2024-06-21T01:52:04.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.two_write_queues: 0 2024-06-21T01:52:04.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.manual_wal_flush: 0 2024-06-21T01:52:04.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.wal_compression: 0 2024-06-21T01:52:04.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.atomic_flush: 0 2024-06-21T01:52:04.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-21T01:52:04.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.log_readahead_size: 0 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.best_efforts_recovery: 0 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.allow_data_in_errors: 0 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.db_host_id: __hostname__ 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_background_jobs: 2 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_background_compactions: -1 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_subcompactions: 1 2024-06-21T01:52:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_total_wal_size: 0 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_open_files: -1 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bytes_per_sync: 0 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_readahead_size: 0 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_background_flushes: -1 2024-06-21T01:52:04.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Compression algorithms supported: 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: kZSTD supported: 0 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: kXpressCompression supported: 0 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: kBZip2Compression supported: 0 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: kLZ4Compression supported: 1 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: kZlibCompression supported: 1 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: kLZ4HCCompression supported: 1 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: kSnappyCompression supported: 1 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.merge_operator: 2024-06-21T01:52:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_filter: None 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_filter_factory: None 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.sst_partitioner_factory: None 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a5269a6d20) 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout: cache_index_and_filter_blocks: 1 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout: pin_top_level_index_and_filter: 1 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_type: 0 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout: data_block_index_type: 0 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_shortening: 1 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout: checksum: 4 2024-06-21T01:52:04.865 INFO:journalctl@ceph.mon.a.smithi092.stdout: no_block_cache: 0 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache: 0x55a52699f1f0 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_name: BinnedLRUCache 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_options: 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: capacity : 536870912 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: num_shard_bits : 4 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: strict_capacity_limit : 0 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: high_pri_pool_ratio: 0.000 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_compressed: (nil) 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: persistent_cache: (nil) 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_size: 4096 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_size_deviation: 10 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_restart_interval: 16 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_block_restart_interval: 1 2024-06-21T01:52:04.866 INFO:journalctl@ceph.mon.a.smithi092.stdout: metadata_block_size: 4096 2024-06-21T01:52:04.867 INFO:journalctl@ceph.mon.a.smithi092.stdout: partition_filters: 0 2024-06-21T01:52:04.867 INFO:journalctl@ceph.mon.a.smithi092.stdout: use_delta_encoding: 1 2024-06-21T01:52:04.867 INFO:journalctl@ceph.mon.a.smithi092.stdout: filter_policy: bloomfilter 2024-06-21T01:52:04.867 INFO:journalctl@ceph.mon.a.smithi092.stdout: whole_key_filtering: 1 2024-06-21T01:52:04.867 INFO:journalctl@ceph.mon.a.smithi092.stdout: verify_compression: 0 2024-06-21T01:52:04.867 INFO:journalctl@ceph.mon.a.smithi092.stdout: read_amp_bytes_per_bit: 0 2024-06-21T01:52:04.867 INFO:journalctl@ceph.mon.a.smithi092.stdout: format_version: 5 2024-06-21T01:52:04.867 INFO:journalctl@ceph.mon.a.smithi092.stdout: enable_index_compression: 1 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_align: 0 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout: max_auto_readahead_size: 262144 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout: prepopulate_block_cache: 0 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout: initial_auto_readahead_size: 8192 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout: num_file_reads_for_auto_readahead: 2 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.write_buffer_size: 33554432 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_write_buffer_number: 2 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compression: NoCompression 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bottommost_compression: Disabled 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.prefix_extractor: nullptr 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.num_levels: 7 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-21T01:52:04.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compression_opts.level: 32767 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compression_opts.strategy: 0 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-21T01:52:04.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-21T01:52:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-21T01:52:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-21T01:52:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compression_opts.enabled: false 2024-06-21T01:52:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-21T01:52:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-21T01:52:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-21T01:52:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-21T01:52:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.target_file_size_base: 67108864 2024-06-21T01:52:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-21T01:52:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.arena_block_size: 1048576 2024-06-21T01:52:04.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-21T01:52:04.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-21T01:52:04.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.disable_auto_compactions: 0 2024-06-21T01:52:04.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-21T01:52:04.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-21T01:52:04.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-21T01:52:04.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-21T01:52:04.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-21T01:52:04.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-21T01:52:04.873 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-21T01:52:04.873 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-21T01:52:04.873 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-21T01:52:04.873 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-21T01:52:04.873 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.table_properties_collectors: 2024-06-21T01:52:04.873 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.inplace_update_support: 0 2024-06-21T01:52:04.873 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-21T01:52:04.873 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-21T01:52:04.873 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-21T01:52:04.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-21T01:52:04.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.bloom_locality: 0 2024-06-21T01:52:04.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.max_successive_merges: 0 2024-06-21T01:52:04.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-21T01:52:04.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.paranoid_file_checks: 0 2024-06-21T01:52:04.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.force_consistency_checks: 1 2024-06-21T01:52:04.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.report_bg_io_stats: 0 2024-06-21T01:52:04.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.ttl: 2592000 2024-06-21T01:52:04.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-21T01:52:04.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-21T01:52:04.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.enable_blob_files: false 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.min_blob_size: 0 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.blob_file_size: 268435456 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.blob_file_starting_level: 0 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-21T01:52:04.875 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12201, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 545ebc10-8111-4c62-890d-5766c47d9cdf 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934724408216, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934724417476, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2470271, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12202, "largest_seqno": 13129, "table_properties": {"data_size": 2465240, "index_size": 2924, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1093, "raw_key_size": 9816, "raw_average_key_size": 23, "raw_value_size": 2456037, "raw_average_value_size": 5847, "num_data_blocks": 134, "num_entries": 420, "num_filter_entries": 420, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1718934724, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "545ebc10-8111-4c62-890d-5766c47d9cdf", "db_session_id": "4MVZDT9W1KNG8J5VN0ZZ", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934724417581, "job": 1, "event": "recovery_finished"} 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a5269cae00 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: DB pointer 0x55a526ac0000 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-21T01:52:04.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** DB Stats ** 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-21T01:52:04.877 INFO:journalctl@ceph.mon.a.smithi092.stdout: L0 1/0 2.36 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 642.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-06-21T01:52:04.878 INFO:journalctl@ceph.mon.a.smithi092.stdout: L6 1/0 7.71 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-06-21T01:52:04.878 INFO:journalctl@ceph.mon.a.smithi092.stdout: Sum 2/0 10.07 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 642.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-06-21T01:52:04.878 INFO:journalctl@ceph.mon.a.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 642.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-06-21T01:52:04.878 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:52:04.878 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-06-21T01:52:04.878 INFO:journalctl@ceph.mon.a.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-21T01:52:04.878 INFO:journalctl@ceph.mon.a.smithi092.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-21T01:52:04.878 INFO:journalctl@ceph.mon.a.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 642.8 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-06-21T01:52:04.878 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:52:04.879 INFO:journalctl@ceph.mon.a.smithi092.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-21T01:52:04.879 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:52:04.879 INFO:journalctl@ceph.mon.a.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-21T01:52:04.879 INFO:journalctl@ceph.mon.a.smithi092.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-06-21T01:52:04.879 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-21T01:52:04.879 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-21T01:52:04.879 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-21T01:52:04.879 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-21T01:52:04.879 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative compaction: 0.00 GB write, 197.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval compaction: 0.00 GB write, 197.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout: Block cache BinnedLRUCache@0x55a52699f1f0#2 capacity: 512.00 MB usage: 4.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.16 KB,0.000220537%) IndexBlock(1,3.08 KB,0.000587106%) Misc(1,0.00 KB,0%) 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: starting mon.a rank 0 at public addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] at bind addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: mon.a@-1(???) e3 preinit fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: mon.a@-1(???).mds e1 new map 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: mon.a@-1(???).mds e1 print_map 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout: e1 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-21T01:52:04.880 INFO:journalctl@ceph.mon.a.smithi092.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-21T01:52:04.881 INFO:journalctl@ceph.mon.a.smithi092.stdout: legacy client fscid: -1 2024-06-21T01:52:04.881 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-06-21T01:52:04.881 INFO:journalctl@ceph.mon.a.smithi092.stdout: No filesystems configured 2024-06-21T01:52:04.881 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: mon.a@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-06-21T01:52:04.881 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-06-21T01:52:04.881 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-06-21T01:52:04.881 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-06-21T01:52:04.881 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 ceph-mon[64276]: mon.a@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-06-21T01:52:04.881 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 bash[64257]: 347e7d52844ca86ab53047fe174c28d99da1cfa0d72b21cfbd244b8734bca3b0 2024-06-21T01:52:04.881 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:04 smithi092 systemd[1]: Started Ceph mon.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:52:06.160 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: ignoring --setuser ceph since I am not root 2024-06-21T01:52:06.160 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: ignoring --setgroup ceph since I am not root 2024-06-21T01:52:06.161 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mgr[54089]: -- 172.21.15.173:0/2370614401 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d1192370e0 con 0x55d119215000 2024-06-21T01:52:06.161 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:06.013+0000 7f2cc17da040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:52:06.161 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:06.163+0000 7f2cc17da040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:52:06.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:06.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 541 B/s rd, 0 op/s 2024-06-21T01:52:06.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:06.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:52:06.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: mon.a calling monitor election 2024-06-21T01:52:06.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-21T01:52:06.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: monmap epoch 3 2024-06-21T01:52:06.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: last_changed 2024-06-21T01:41:49.138463+0000 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: min_mon_release 17 (quincy) 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: election_strategy: 1 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: fsmap 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: osdmap e105: 8 total, 8 up, 8 in 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: mgrmap e33: y(active, since 33s), standbys: x 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: daemon iscsi.foo.smithi092.pqnhsu on smithi092 is in unknown state 2024-06-21T01:52:06.162 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: from='mgr.25123 ' entity='' 2024-06-21T01:52:06.163 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:05 smithi173 ceph-mon[25952]: mgrmap e34: y(active, since 33s), standbys: x 2024-06-21T01:52:06.164 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ignoring --setuser ceph since I am not root 2024-06-21T01:52:06.165 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ignoring --setgroup ceph since I am not root 2024-06-21T01:52:06.165 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mgr[60004]: -- 172.21.15.92:0/1562781215 <== mon.1 v2:172.21.15.92:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dece8d50e0 con 0x55dece8b3400 2024-06-21T01:52:06.165 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:06 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:06.017+0000 7f99c681d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-21T01:52:06.165 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:06 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:06.167+0000 7f99c681d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:52:06.165 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:06.165 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 541 B/s rd, 0 op/s 2024-06-21T01:52:06.165 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:06.165 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:52:06.165 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: mon.a calling monitor election 2024-06-21T01:52:06.165 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: monmap epoch 3 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: last_changed 2024-06-21T01:41:49.138463+0000 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: min_mon_release 17 (quincy) 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: election_strategy: 1 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: fsmap 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: osdmap e105: 8 total, 8 up, 8 in 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: mgrmap e33: y(active, since 33s), standbys: x 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-06-21T01:52:06.166 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-21T01:52:06.167 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: daemon iscsi.foo.smithi092.pqnhsu on smithi092 is in unknown state 2024-06-21T01:52:06.167 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: from='mgr.25123 ' entity='' 2024-06-21T01:52:06.167 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[64276]: mgrmap e34: y(active, since 33s), standbys: x 2024-06-21T01:52:06.167 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: from='mgr.25123 172.21.15.92:0/1868765411' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:06.167 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 541 B/s rd, 0 op/s 2024-06-21T01:52:06.167 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:06.167 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:52:06.167 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: mon.a calling monitor election 2024-06-21T01:52:06.167 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-21T01:52:06.167 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: monmap epoch 3 2024-06-21T01:52:06.167 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:06.168 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: last_changed 2024-06-21T01:41:49.138463+0000 2024-06-21T01:52:06.168 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:06.168 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: min_mon_release 17 (quincy) 2024-06-21T01:52:06.168 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: election_strategy: 1 2024-06-21T01:52:06.168 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:06.168 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:06.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:06.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: fsmap 2024-06-21T01:52:06.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: osdmap e105: 8 total, 8 up, 8 in 2024-06-21T01:52:06.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: mgrmap e33: y(active, since 33s), standbys: x 2024-06-21T01:52:06.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-06-21T01:52:06.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-21T01:52:06.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: daemon iscsi.foo.smithi092.pqnhsu on smithi092 is in unknown state 2024-06-21T01:52:06.170 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: from='mgr.25123 ' entity='' 2024-06-21T01:52:06.170 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:05 smithi092 ceph-mon[29810]: mgrmap e34: y(active, since 33s), standbys: x 2024-06-21T01:52:06.477 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:06.233+0000 7f2cc17da040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:52:06.478 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:06 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:06.237+0000 7f99c681d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-21T01:52:06.478 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:06 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:06.480+0000 7f99c681d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:52:06.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:06.480+0000 7f2cc17da040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-21T01:52:06.728 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:06.544+0000 7f2cc17da040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:52:06.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:06 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:06.544+0000 7f99c681d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-21T01:52:07.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:07 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:07.505+0000 7f99c681d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:52:07.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:07 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:07.631+0000 7f99c681d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:52:07.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:07 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:07.693+0000 7f99c681d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:52:07.855 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:07.494+0000 7f2cc17da040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-21T01:52:07.855 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:07.613+0000 7f2cc17da040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-21T01:52:07.855 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:07.671+0000 7f2cc17da040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-21T01:52:08.168 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:07 smithi173 ceph-mon[25952]: mgrmap e35: y(active, since 34s), standbys: x 2024-06-21T01:52:08.168 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:07.891+0000 7f2cc17da040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:52:08.168 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:07.954+0000 7f2cc17da040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:52:08.168 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:08.021+0000 7f2cc17da040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:52:08.168 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:08.171+0000 7f2cc17da040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:52:08.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:07 smithi092 ceph-mon[64276]: mgrmap e35: y(active, since 34s), standbys: x 2024-06-21T01:52:08.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:07 smithi092 ceph-mon[29810]: mgrmap e35: y(active, since 34s), standbys: x 2024-06-21T01:52:08.214 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:07 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:07.926+0000 7f99c681d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-21T01:52:08.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:07 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:07.991+0000 7f99c681d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-21T01:52:08.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:08 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:08.062+0000 7f99c681d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-21T01:52:08.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:08 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:08.217+0000 7f99c681d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-21T01:52:08.919 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:08.597+0000 7f2cc17da040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:52:08.919 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:08.922+0000 7f2cc17da040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:52:08.920 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:52:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:52:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:52:08.976 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:08 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:08.652+0000 7f99c681d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-21T01:52:09.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:08.990+0000 7f2cc17da040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:52:09.228 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:09.143+0000 7f2cc17da040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-21T01:52:09.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:08 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:08.979+0000 7f99c681d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-21T01:52:09.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:09.047+0000 7f99c681d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-21T01:52:09.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:09.203+0000 7f99c681d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-21T01:52:09.726 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:09.465+0000 7f2cc17da040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:52:09.726 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:09.531+0000 7f2cc17da040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:52:09.773 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:09.523+0000 7f99c681d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-21T01:52:09.773 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:09.585+0000 7f99c681d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-21T01:52:09.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:09.727+0000 7f2cc17da040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:52:09.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:09.881+0000 7f2cc17da040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:52:09.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:09.951+0000 7f2cc17da040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:52:10.074 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:09.776+0000 7f99c681d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-21T01:52:10.074 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:09.928+0000 7f99c681d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-21T01:52:10.074 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:09.993+0000 7f99c681d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-21T01:52:10.074 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:10.077+0000 7f99c681d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:52:10.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:10.138+0000 7f99c681d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:52:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:10.039+0000 7f2cc17da040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-21T01:52:10.478 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:10.099+0000 7f2cc17da040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-21T01:52:10.912 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: Standby manager daemon x restarted 2024-06-21T01:52:10.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: Standby manager daemon x started 2024-06-21T01:52:10.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:52:10.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:52:10.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:52:10.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:52:10.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: Active manager daemon y restarted 2024-06-21T01:52:10.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: Activating manager daemon y 2024-06-21T01:52:10.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:10.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: mgrmap e36: y(active, starting, since 0.00782666s), standbys: x 2024-06-21T01:52:10.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:52:10.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:52:10.914 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:52:10.914 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:52:10.914 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:10.645+0000 7f2cc17da040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:52:10.914 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:10.809+0000 7f2cc17da040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:52:10.914 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: [21/Jun/2024:01:52:10] ENGINE Bus STARTING 2024-06-21T01:52:10.914 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: CherryPy Checker: 2024-06-21T01:52:10.914 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: The Application mounted at '' has an empty config. 2024-06-21T01:52:10.914 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: 2024-06-21T01:52:10.914 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: [21/Jun/2024:01:52:10] ENGINE Serving on http://:::9283 2024-06-21T01:52:10.914 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 01:52:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x[54085]: [21/Jun/2024:01:52:10] ENGINE Bus STARTED 2024-06-21T01:52:10.931 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:10.684+0000 7f99c681d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-21T01:52:10.931 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:10.847+0000 7f99c681d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-21T01:52:10.931 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:52:10] ENGINE Bus STARTING 2024-06-21T01:52:10.931 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: CherryPy Checker: 2024-06-21T01:52:10.931 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: The Application mounted at '' has an empty config. 2024-06-21T01:52:10.931 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: Standby manager daemon x restarted 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: Standby manager daemon x started 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: Active manager daemon y restarted 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: Activating manager daemon y 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: mgrmap e36: y(active, starting, since 0.00782666s), standbys: x 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:52:10.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:52:10.933 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:52:10.933 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:52:10.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: Standby manager daemon x restarted 2024-06-21T01:52:10.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: Standby manager daemon x started 2024-06-21T01:52:10.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-21T01:52:10.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-21T01:52:10.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-21T01:52:10.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: from='mgr.? 172.21.15.173:0/3675928244' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-21T01:52:10.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: Active manager daemon y restarted 2024-06-21T01:52:10.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: Activating manager daemon y 2024-06-21T01:52:10.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:10.934 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: mgrmap e36: y(active, starting, since 0.00782666s), standbys: x 2024-06-21T01:52:10.934 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:52:10.934 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:52:10.934 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:52:10.934 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:10 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-21T01:52:11.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:52:11] ENGINE Serving on http://:::9283 2024-06-21T01:52:11.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:52:11] ENGINE Bus STARTED 2024-06-21T01:52:11.896 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:52:12.118 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:11.882+0000 7f9965fe7640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-06-21T01:52:12.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:52:12.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:52:12.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:52:12.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:52:12.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:52:12.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:52:12.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:52:12.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:52:12.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:52:12.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: Manager daemon y is now available 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:52:12.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: Manager daemon y is now available 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:52:12.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:52:12.122 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:11 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:52:12.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-21T01:52:12.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:52:12.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:52:12.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:52:12.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:52:12.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:52:12.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:52:12.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:52:12.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:52:12.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-21T01:52:12.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-21T01:52:12.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-21T01:52:12.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: Manager daemon y is now available 2024-06-21T01:52:12.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:12.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:52:12.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-21T01:52:12.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:52:12.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:11 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-21T01:52:12.606 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (71s) 35s ago 5m 14.7M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (57s) 17s ago 5m 46.4M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 unknown 35s ago 4m - - 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (22s) 17s ago 10m 53.9M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (52s) 35s ago 11m 473M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (11m) 35s ago 11m 56.4M 2048M 17.2.0 e1d6a67b021e 3daa4a891e57 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (10m) 17s ago 10m 48.3M 2048M 17.2.0 e1d6a67b021e 00fcf1f13930 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (10m) 35s ago 10m 45.1M 2048M 17.2.0 e1d6a67b021e 6a24d6b0bbb6 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (76s) 35s ago 6m 8468k - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (61s) 17s ago 6m 14.8M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (9m) 35s ago 9m 52.8M 2736M 17.2.0 e1d6a67b021e 63507222842b 2024-06-21T01:52:12.607 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (9m) 35s ago 9m 54.7M 2736M 17.2.0 e1d6a67b021e e0bf7c0600e5 2024-06-21T01:52:12.608 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (9m) 35s ago 9m 47.7M 2736M 17.2.0 e1d6a67b021e 157fb16866c9 2024-06-21T01:52:12.608 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (8m) 35s ago 8m 51.2M 2736M 17.2.0 e1d6a67b021e e5567e44eaec 2024-06-21T01:52:12.608 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (8m) 17s ago 8m 55.1M 2992M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:52:12.608 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (7m) 17s ago 7m 53.3M 2992M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:52:12.608 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (7m) 17s ago 7m 48.3M 2992M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:52:12.608 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (7m) 17s ago 7m 52.5M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:52:12.608 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (26s) 17s ago 6m 48.5M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:52:12.608 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (4m) 35s ago 4m 74.8M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:52:12.608 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (4m) 17s ago 4m 72.0M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:52:12.608 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (4m) 35s ago 4m 72.2M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:52:12.608 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (4m) 17s ago 4m 73.1M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:52:12.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:12.546+0000 7f9923b5b640 -1 log_channel(cephadm) log [ERR] : [21/Jun/2024:01:52:12] ENGINE Error in HTTPServer.serve 2024-06-21T01:52:12.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: Traceback (most recent call last): 2024-06-21T01:52:12.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:52:12.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: self._connections.run(self.expiration_interval) 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: self._run(expiration_interval) 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: s = self.context.wrap_socket( 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: return self.sslsocket_class._create( 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:52:12.852 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: self.do_handshake() 2024-06-21T01:52:12.853 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:52:12.853 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: self._sslobj.do_handshake() 2024-06-21T01:52:12.853 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:52:12.853 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:12.882 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:12 smithi173 ceph-mon[25952]: from='client.34113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:12.882 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:12 smithi173 ceph-mon[25952]: mgrmap e37: y(active, since 1.02751s), standbys: x 2024-06-21T01:52:12.882 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:12 smithi173 ceph-mon[25952]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:13.041 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:52:13.041 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:52:13.041 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-06-21T01:52:13.041 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 1 2024-06-21T01:52:13.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:52:13.041 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:52:13.041 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:52:13.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:52:13.042 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:52:13.042 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-21T01:52:13.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:52:13.042 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:52:13.042 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:52:13.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:52:13.042 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:52:13.042 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-06-21T01:52:13.042 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:52:13.042 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:52:13.042 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:52:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-mon[64276]: from='client.34113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-mon[64276]: mgrmap e37: y(active, since 1.02751s), standbys: x 2024-06-21T01:52:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-mon[64276]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:13.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-mon[29810]: from='client.34113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:13.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-mon[29810]: mgrmap e37: y(active, since 1.02751s), standbys: x 2024-06-21T01:52:13.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:12 smithi092 ceph-mon[29810]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:13.441 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:52:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:52:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:52:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:52:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:52:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr" 2024-06-21T01:52:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:52:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "2/23 daemons upgraded", 2024-06-21T01:52:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "", 2024-06-21T01:52:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:52:13.442 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:52:13.786 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:52:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:52:13.842 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:52:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: [21/Jun/2024:01:52:12] ENGINE Bus STARTING 2024-06-21T01:52:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: [21/Jun/2024:01:52:12] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:52:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: [21/Jun/2024:01:52:12] ENGINE Error in HTTPServer.serve 2024-06-21T01:52:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:52:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:52:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout: self._connections.run(self.expiration_interval) 2024-06-21T01:52:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:52:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout: self._run(expiration_interval) 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: s = self.context.wrap_socket( 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: return self.sslsocket_class._create( 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: self.do_handshake() 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: self._sslobj.do_handshake() 2024-06-21T01:52:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:52:14.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:14.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: [21/Jun/2024:01:52:12] ENGINE Serving on http://172.21.15.92:8765 2024-06-21T01:52:14.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: [21/Jun/2024:01:52:12] ENGINE Bus STARTED 2024-06-21T01:52:14.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:14.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-06-21T01:52:14.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: Cluster is now healthy 2024-06-21T01:52:14.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: mgrmap e38: y(active, since 2s), standbys: x 2024-06-21T01:52:14.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/2415355639' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:14.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/3280324675' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:52:14.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: [21/Jun/2024:01:52:12] ENGINE Bus STARTING 2024-06-21T01:52:14.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: [21/Jun/2024:01:52:12] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: [21/Jun/2024:01:52:12] ENGINE Error in HTTPServer.serve 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: Traceback (most recent call last): 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: self._connections.run(self.expiration_interval) 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: self._run(expiration_interval) 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: s = self.context.wrap_socket( 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: return self.sslsocket_class._create( 2024-06-21T01:52:14.104 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout: self.do_handshake() 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout: self._sslobj.do_handshake() 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: [21/Jun/2024:01:52:12] ENGINE Serving on http://172.21.15.92:8765 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: [21/Jun/2024:01:52:12] ENGINE Bus STARTED 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: Cluster is now healthy 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: mgrmap e38: y(active, since 2s), standbys: x 2024-06-21T01:52:14.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/2415355639' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:14.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:13 smithi092 ceph-mon[29810]: from='client.? 172.21.15.92:0/3280324675' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:52:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: [21/Jun/2024:01:52:12] ENGINE Bus STARTING 2024-06-21T01:52:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: [21/Jun/2024:01:52:12] ENGINE Serving on https://172.21.15.92:7150 2024-06-21T01:52:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: [21/Jun/2024:01:52:12] ENGINE Error in HTTPServer.serve 2024-06-21T01:52:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout: Traceback (most recent call last): 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._connections.run(self.expiration_interval) 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._run(expiration_interval) 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: s = self.context.wrap_socket( 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: return self.sslsocket_class._create( 2024-06-21T01:52:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout: self.do_handshake() 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout: self._sslobj.do_handshake() 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: [21/Jun/2024:01:52:12] ENGINE Serving on http://172.21.15.92:8765 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: [21/Jun/2024:01:52:12] ENGINE Bus STARTED 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: Cluster is now healthy 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: mgrmap e38: y(active, since 2s), standbys: x 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/2415355639' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:13 smithi173 ceph-mon[25952]: from='client.? 172.21.15.92:0/3280324675' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:52:15.145 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:14 smithi092 ceph-mon[64276]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:15.145 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:14 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:15.145 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:14 smithi092 ceph-mon[29810]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:15.145 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:14 smithi092 ceph-mon[29810]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:15.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:14 smithi173 ceph-mon[25952]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:15.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:14 smithi173 ceph-mon[25952]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:15 smithi173 ceph-mon[25952]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:15 smithi173 ceph-mon[25952]: mgrmap e39: y(active, since 4s), standbys: x 2024-06-21T01:52:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:15 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:15 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:15 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:16.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:15 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[64276]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[64276]: mgrmap e39: y(active, since 4s), standbys: x 2024-06-21T01:52:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:16.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[29810]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:16.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[29810]: mgrmap e39: y(active, since 4s), standbys: x 2024-06-21T01:52:16.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:16.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:16.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:16.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:15 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:17 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:17 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:17 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:17 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:17 smithi092 ceph-mon[64276]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:17 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:17 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:17 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:17 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:17 smithi092 ceph-mon[29810]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:17.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:17 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:17 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:17 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:17 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:17.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:17 smithi173 ceph-mon[25952]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:52:19.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:52:19] "GET /metrics HTTP/1.1" 200 35449 "" "Prometheus/2.33.4" 2024-06-21T01:52:20.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:19 smithi173 ceph-mon[25952]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:52:20.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:19 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:20.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:19 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:20.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:19 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:20.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:19 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:19 smithi092 ceph-mon[64276]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:52:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:19 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:19 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:19 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:19 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:20.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:19 smithi092 ceph-mon[29810]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-21T01:52:20.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:19 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:20.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:19 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:20.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:19 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:20.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:19 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:21.043 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:52:21.043 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:52:21.043 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:52:21.043 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-06-21T01:52:21.044 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:21.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-21T01:52:21.046 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:20 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-06-21T01:52:21.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: Updating smithi092:/etc/ceph/ceph.conf 2024-06-21T01:52:21.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-21T01:52:21.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:52:21.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-21T01:52:21.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:52:21.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-06-21T01:52:21.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:21.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:52:21.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:21.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-21T01:52:21.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:20 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-06-21T01:52:22.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[64276]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:52:22.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[64276]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:52:22.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[64276]: Upgrade: It appears safe to stop mon.c 2024-06-21T01:52:22.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[64276]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-21T01:52:22.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[64276]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:22.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:22.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:22.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:22.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[29810]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:52:22.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[29810]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:52:22.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[29810]: Upgrade: It appears safe to stop mon.c 2024-06-21T01:52:22.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[29810]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-21T01:52:22.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[29810]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:22.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:22.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:22.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:21 smithi092 ceph-mon[29810]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:22.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:21 smithi173 ceph-mon[25952]: Updating smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:52:22.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:21 smithi173 ceph-mon[25952]: Updating smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-21T01:52:22.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:21 smithi173 ceph-mon[25952]: Upgrade: It appears safe to stop mon.c 2024-06-21T01:52:22.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:21 smithi173 ceph-mon[25952]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-21T01:52:22.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:21 smithi173 ceph-mon[25952]: from='mgr.34107 ' entity='mgr.y' 2024-06-21T01:52:22.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:21 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:22.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:21 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:22.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:21 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:23.198 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:22 smithi092 ceph-mon[64276]: Upgrade: Updating mon.c 2024-06-21T01:52:23.199 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:22 smithi092 ceph-mon[64276]: Deploying daemon mon.c on smithi092 2024-06-21T01:52:23.199 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:22 smithi092 ceph-mon[29810]: Upgrade: Updating mon.c 2024-06-21T01:52:23.199 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:22 smithi092 ceph-mon[29810]: Deploying daemon mon.c on smithi092 2024-06-21T01:52:23.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:22 smithi173 ceph-mon[25952]: Upgrade: Updating mon.c 2024-06-21T01:52:23.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:22 smithi173 ceph-mon[25952]: Deploying daemon mon.c on smithi092 2024-06-21T01:52:23.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:23 smithi092 systemd[1]: Stopping Ceph mon.c for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:52:23.894 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c[29806]: 2024-06-21T01:52:23.639+0000 7f18d2514700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:52:23.895 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c[29806]: 2024-06-21T01:52:23.639+0000 7f18d2514700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-06-21T01:52:23.895 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:23 smithi092 podman[66587]: 2024-06-21 01:52:23.660806629 +0000 UTC m=+0.120314245 container died 6a24d6b0bbb6aeaf135833a0970d74df90ce8ce8511e75a3a240db9d13ef3f2f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-type=git, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, release=754, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-06-21T01:52:23.895 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:52:23] ENGINE Bus STOPPING 2024-06-21T01:52:23.895 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:52:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-21T01:52:23.895 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:52:23] ENGINE Bus STOPPED 2024-06-21T01:52:23.895 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:52:23] ENGINE Bus STARTING 2024-06-21T01:52:23.895 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:52:23] ENGINE Serving on http://:::9283 2024-06-21T01:52:23.895 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:01:52:23] ENGINE Bus STARTED 2024-06-21T01:52:23.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:52:23 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:52:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:52:24.179 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:23 smithi092 podman[66587]: 2024-06-21 01:52:23.997837045 +0000 UTC m=+0.457344653 container cleanup 6a24d6b0bbb6aeaf135833a0970d74df90ce8ce8511e75a3a240db9d13ef3f2f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_BRANCH=HEAD, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, name=centos-stream, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-06-21T01:52:24.179 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:23 smithi092 bash[66587]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c 2024-06-21T01:52:24.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:24 smithi092 podman[66601]: 2024-06-21 01:52:24.182991236 +0000 UTC m=+0.521852361 container remove 6a24d6b0bbb6aeaf135833a0970d74df90ce8ce8511e75a3a240db9d13ef3f2f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-06-21T01:52:24.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:24 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.c.service: Deactivated successfully. 2024-06-21T01:52:24.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:24 smithi092 systemd[1]: Stopped Ceph mon.c for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:52:24.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:24 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.c.service: Consumed 7.741s CPU time. 2024-06-21T01:52:24.839 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:24 smithi092 systemd[1]: Starting Ceph mon.c for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:52:25.099 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:24 smithi092 podman[66737]: 2024-06-21 01:52:24.746117882 +0000 UTC m=+0.017459271 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:52:25.099 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:24 smithi092 podman[66737]: 2024-06-21 01:52:24.892232127 +0000 UTC m=+0.163573509 container create 6549ab7d4df8ea76b1ae50cfe548d29200ca0a7299992d52aafc729a0fd5222d (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-21T01:52:25.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[64276]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-21T01:52:25.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:52:25.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:52:25.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:52:25.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 podman[66737]: 2024-06-21 01:52:25.102981389 +0000 UTC m=+0.374322777 container init 6549ab7d4df8ea76b1ae50cfe548d29200ca0a7299992d52aafc729a0fd5222d (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-06-21T01:52:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 podman[66737]: 2024-06-21 01:52:25.107082513 +0000 UTC m=+0.378423906 container start 6549ab7d4df8ea76b1ae50cfe548d29200ca0a7299992d52aafc729a0fd5222d (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, GIT_CLEAN=True) 2024-06-21T01:52:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: set uid:gid to 167:167 (ceph:ceph) 2024-06-21T01:52:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc), process ceph-mon, pid 2 2024-06-21T01:52:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: pidfile_write: ignore empty --pid-file 2024-06-21T01:52:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: load: jerasure load: lrc 2024-06-21T01:52:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: RocksDB version: 7.9.2 2024-06-21T01:52:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Git sha 0 2024-06-21T01:52:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Compile date 2024-06-19 15:40:36 2024-06-21T01:52:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: DB SUMMARY 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: DB Session ID: IWYEOL15E8H196UN2MYO 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: CURRENT file: CURRENT 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: IDENTITY file: IDENTITY 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 285430 ; 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.error_if_exists: 0 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.create_if_missing: 0 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.paranoid_checks: 1 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.env: 0x561955c01740 2024-06-21T01:52:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.fs: PosixFileSystem 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.info_log: 0x561956feb0c0 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_file_opening_threads: 16 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.statistics: (nil) 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.use_fsync: 0 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_log_file_size: 0 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.keep_log_file_num: 1000 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.recycle_log_file_num: 0 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.allow_fallocate: 1 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.allow_mmap_reads: 0 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.allow_mmap_writes: 0 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.use_direct_reads: 0 2024-06-21T01:52:25.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.create_missing_column_families: 0 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.db_log_dir: 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.wal_dir: 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.advise_random_on_open: 1 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.db_write_buffer_size: 0 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.write_buffer_manager: 0x561956ffac80 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-21T01:52:25.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.rate_limiter: (nil) 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.wal_recovery_mode: 2 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.enable_thread_tracking: 0 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.enable_pipelined_write: 0 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.unordered_write: 0 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.row_cache: None 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.wal_filter: None 2024-06-21T01:52:25.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-21T01:52:25.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.allow_ingest_behind: 0 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.two_write_queues: 0 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.manual_wal_flush: 0 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.wal_compression: 0 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.atomic_flush: 0 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.log_readahead_size: 0 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.best_efforts_recovery: 0 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.allow_data_in_errors: 0 2024-06-21T01:52:25.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.db_host_id: __hostname__ 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_background_jobs: 2 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_background_compactions: -1 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_subcompactions: 1 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_total_wal_size: 0 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_open_files: -1 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bytes_per_sync: 0 2024-06-21T01:52:25.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_readahead_size: 0 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_background_flushes: -1 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Compression algorithms supported: 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: kZSTD supported: 0 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: kXpressCompression supported: 0 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: kBZip2Compression supported: 0 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: kLZ4Compression supported: 1 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: kZlibCompression supported: 1 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: kLZ4HCCompression supported: 1 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: kSnappyCompression supported: 1 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-21T01:52:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.merge_operator: 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_filter: None 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_filter_factory: None 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.sst_partitioner_factory: None 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561956fead20) 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout: cache_index_and_filter_blocks: 1 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout: pin_top_level_index_and_filter: 1 2024-06-21T01:52:25.361 INFO:journalctl@ceph.mon.c.smithi092.stdout: index_type: 0 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: data_block_index_type: 0 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: index_shortening: 1 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: checksum: 4 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: no_block_cache: 0 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_cache: 0x561956fe31f0 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_cache_name: BinnedLRUCache 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_cache_options: 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: capacity : 536870912 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: num_shard_bits : 4 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: strict_capacity_limit : 0 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: high_pri_pool_ratio: 0.000 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_cache_compressed: (nil) 2024-06-21T01:52:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout: persistent_cache: (nil) 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_size: 4096 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_size_deviation: 10 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_restart_interval: 16 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: index_block_restart_interval: 1 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: metadata_block_size: 4096 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: partition_filters: 0 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: use_delta_encoding: 1 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: filter_policy: bloomfilter 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: whole_key_filtering: 1 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: verify_compression: 0 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: read_amp_bytes_per_bit: 0 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: format_version: 5 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: enable_index_compression: 1 2024-06-21T01:52:25.363 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_align: 0 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout: max_auto_readahead_size: 262144 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout: prepopulate_block_cache: 0 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout: initial_auto_readahead_size: 8192 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout: num_file_reads_for_auto_readahead: 2 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.write_buffer_size: 33554432 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_write_buffer_number: 2 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compression: NoCompression 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bottommost_compression: Disabled 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.prefix_extractor: nullptr 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.num_levels: 7 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-21T01:52:25.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compression_opts.level: 32767 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compression_opts.strategy: 0 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-21T01:52:25.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compression_opts.enabled: false 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.target_file_size_base: 67108864 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-21T01:52:25.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.arena_block_size: 1048576 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.disable_auto_compactions: 0 2024-06-21T01:52:25.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.table_properties_collectors: 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.inplace_update_support: 0 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-21T01:52:25.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.bloom_locality: 0 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.max_successive_merges: 0 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.paranoid_file_checks: 0 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.force_consistency_checks: 1 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.report_bg_io_stats: 0 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.ttl: 2592000 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.enable_blob_files: false 2024-06-21T01:52:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.min_blob_size: 0 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.blob_file_size: 268435456 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.blob_file_starting_level: 0 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14545, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d36499f5-5e59-46a4-aee0-d18199e6dd8f 2024-06-21T01:52:25.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934745133485, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934745134909, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 190426, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14550, "largest_seqno": 14671, "table_properties": {"data_size": 188738, "index_size": 484, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 197, "raw_key_size": 1903, "raw_average_key_size": 26, "raw_value_size": 186991, "raw_average_value_size": 2633, "num_data_blocks": 19, "num_entries": 71, "num_filter_entries": 71, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1718934745, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d36499f5-5e59-46a4-aee0-d18199e6dd8f", "db_session_id": "IWYEOL15E8H196UN2MYO", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934745135003, "job": 1, "event": "recovery_finished"} 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56195700ee00 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: DB pointer 0x561957104000 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** DB Stats ** 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-21T01:52:25.371 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** Compaction Stats [default] ** 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: L0 1/0 185.96 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 200.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: L6 1/0 10.66 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: Sum 2/0 10.85 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 200.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 200.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-06-21T01:52:25.372 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** Compaction Stats [default] ** 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 200.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative compaction: 0.00 GB write, 51.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval compaction: 0.00 GB write, 51.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-21T01:52:25.373 INFO:journalctl@ceph.mon.c.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout: Block cache BinnedLRUCache@0x561956fe31f0#2 capacity: 512.00 MB usage: 0.81 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.23 KB,4.47035e-05%) IndexBlock(1,0.58 KB,0.000110269%) Misc(1,0.00 KB,0%) 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: starting mon.c rank 1 at public addrs [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] at bind addrs [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: mon.c@-1(???) e3 preinit fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: mon.c@-1(???).mds e1 new map 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: mon.c@-1(???).mds e1 print_map 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout: e1 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout: legacy client fscid: -1 2024-06-21T01:52:25.374 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout: No filesystems configured 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: mon.c@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: mon.c@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 bash[66737]: 6549ab7d4df8ea76b1ae50cfe548d29200ca0a7299992d52aafc729a0fd5222d 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 systemd[1]: Started Ceph mon.c for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:52:25.375 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:25 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:25.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:25 smithi173 ceph-mon[25952]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-21T01:52:25.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:25 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-21T01:52:25.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:25 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-21T01:52:25.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:25 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-21T01:52:25.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:25 smithi173 ceph-mon[25952]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:26.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:52:26.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: mon.c calling monitor election 2024-06-21T01:52:26.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: mon.a calling monitor election 2024-06-21T01:52:26.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-21T01:52:26.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: monmap epoch 3 2024-06-21T01:52:26.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:26.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: last_changed 2024-06-21T01:41:49.138463+0000 2024-06-21T01:52:26.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: min_mon_release 17 (quincy) 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: election_strategy: 1 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: fsmap 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: mgrmap e39: y(active, since 14s), standbys: x 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: overall HEALTH_OK 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:26.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: mon.c calling monitor election 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: mon.a calling monitor election 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: monmap epoch 3 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: last_changed 2024-06-21T01:41:49.138463+0000 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: min_mon_release 17 (quincy) 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: election_strategy: 1 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:26.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: fsmap 2024-06-21T01:52:26.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:26.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: mgrmap e39: y(active, since 14s), standbys: x 2024-06-21T01:52:26.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: overall HEALTH_OK 2024-06-21T01:52:26.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:26.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:26.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:26.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:26.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:52:26.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: mon.c calling monitor election 2024-06-21T01:52:26.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: mon.a calling monitor election 2024-06-21T01:52:26.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: monmap epoch 3 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: last_changed 2024-06-21T01:41:49.138463+0000 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: min_mon_release 17 (quincy) 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: election_strategy: 1 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: fsmap 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: mgrmap e39: y(active, since 14s), standbys: x 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: overall HEALTH_OK 2024-06-21T01:52:26.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:26.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:26.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:26.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:26 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:27.556 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:27 smithi092 ceph-mon[64276]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:52:27.557 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:27 smithi092 ceph-mon[66756]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:52:27.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:27 smithi173 ceph-mon[25952]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:52:28.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:52:28 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:52:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:52:29.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:52:29] "GET /metrics HTTP/1.1" 200 38188 "" "Prometheus/2.33.4" 2024-06-21T01:52:30.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:29 smithi173 ceph-mon[25952]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:52:30.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:29 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:30.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:29 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:30.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:29 smithi092 ceph-mon[64276]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:52:30.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:29 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:30.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:29 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:30.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:29 smithi092 ceph-mon[66756]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-21T01:52:30.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:29 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:30.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:29 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:30.333 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:30 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:30.145+0000 7f9965fe7640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-06-21T01:52:31.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:30 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:31.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:30 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:31.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:30 smithi173 ceph-mon[25952]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:31.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:30 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:31.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:30 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:31.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:30 smithi092 ceph-mon[64276]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:31.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:30 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:31.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:30 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:31.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:30 smithi092 ceph-mon[66756]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:33.849 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:52:33 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:52:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:52:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[64276]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:52:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:52:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-21T01:52:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-06-21T01:52:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[66756]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:52:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:52:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-21T01:52:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-06-21T01:52:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:33 smithi173 ceph-mon[25952]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:52:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:33 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:33 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:33 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:33 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:33 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:33 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:52:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:33 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:33 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-21T01:52:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:33 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-06-21T01:52:34.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:34 smithi173 ceph-mon[25952]: Upgrade: It appears safe to stop mon.b 2024-06-21T01:52:34.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:34 smithi173 ceph-mon[25952]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:34.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:34 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:34.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:34 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:34.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:34 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:34.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:34 smithi173 ceph-mon[25952]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[66756]: Upgrade: It appears safe to stop mon.b 2024-06-21T01:52:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:35.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[64276]: Upgrade: It appears safe to stop mon.b 2024-06-21T01:52:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:34 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:36.030 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:35 smithi173 ceph-mon[25952]: Upgrade: Updating mon.b 2024-06-21T01:52:36.031 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:35 smithi173 ceph-mon[25952]: Deploying daemon mon.b on smithi173 2024-06-21T01:52:36.031 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:35 smithi173 ceph-mon[25952]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:36.252 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:35 smithi092 ceph-mon[64276]: Upgrade: Updating mon.b 2024-06-21T01:52:36.252 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:35 smithi092 ceph-mon[64276]: Deploying daemon mon.b on smithi173 2024-06-21T01:52:36.253 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:35 smithi092 ceph-mon[64276]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:36.253 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:35 smithi092 ceph-mon[66756]: Upgrade: Updating mon.b 2024-06-21T01:52:36.253 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:35 smithi092 ceph-mon[66756]: Deploying daemon mon.b on smithi173 2024-06-21T01:52:36.253 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:35 smithi092 ceph-mon[66756]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:36.594 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:36 smithi173 systemd[1]: Stopping Ceph mon.b for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:52:36.920 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b[25936]: 2024-06-21T01:52:36.596+0000 7f1dd5818700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:52:36.920 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b[25936]: 2024-06-21T01:52:36.596+0000 7f1dd5818700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-06-21T01:52:36.920 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:36 smithi173 podman[57383]: 2024-06-21 01:52:36.618206436 +0000 UTC m=+0.151525429 container died 00fcf1f139305e6a7d7f9575d0b7ef3ef18e9d1ab80883f7e044642dd2916d94 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b, ceph=True, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, architecture=x86_64, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., version=8) 2024-06-21T01:52:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:37 smithi173 podman[57383]: 2024-06-21 01:52:37.014463798 +0000 UTC m=+0.547782791 container cleanup 00fcf1f139305e6a7d7f9575d0b7ef3ef18e9d1ab80883f7e044642dd2916d94 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, vcs-type=git, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, distribution-scope=public, ceph=True, version=8, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-06-21T01:52:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:37 smithi173 bash[57383]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b 2024-06-21T01:52:37.588 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:37 smithi173 podman[57395]: 2024-06-21 01:52:37.257600392 +0000 UTC m=+0.637357502 container remove 00fcf1f139305e6a7d7f9575d0b7ef3ef18e9d1ab80883f7e044642dd2916d94 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, name=centos-stream, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, distribution-scope=public, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True) 2024-06-21T01:52:37.588 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:37 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.b.service: Deactivated successfully. 2024-06-21T01:52:37.588 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:37 smithi173 systemd[1]: Stopped Ceph mon.b for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:52:37.588 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:37 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.b.service: Consumed 9.028s CPU time. 2024-06-21T01:52:37.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:37 smithi173 systemd[1]: Starting Ceph mon.b for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:52:37.839 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:37 smithi173 podman[57521]: 2024-06-21 01:52:37.709309053 +0000 UTC m=+0.017004395 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:52:38.100 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:37 smithi173 podman[57521]: 2024-06-21 01:52:37.84261845 +0000 UTC m=+0.150313782 container create e6b535ec7a5546c1152bd41779ca2ac70e3b843a76e95b570d26167fedb362ca (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-21T01:52:38.100 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 podman[57521]: 2024-06-21 01:52:38.078763043 +0000 UTC m=+0.386458385 container init e6b535ec7a5546c1152bd41779ca2ac70e3b843a76e95b570d26167fedb362ca (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:52:38.100 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 podman[57521]: 2024-06-21 01:52:38.082273272 +0000 UTC m=+0.389968605 container start e6b535ec7a5546c1152bd41779ca2ac70e3b843a76e95b570d26167fedb362ca (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-06-21T01:52:38.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: set uid:gid to 167:167 (ceph:ceph) 2024-06-21T01:52:38.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc), process ceph-mon, pid 2 2024-06-21T01:52:38.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: pidfile_write: ignore empty --pid-file 2024-06-21T01:52:38.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: load: jerasure load: lrc 2024-06-21T01:52:38.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: RocksDB version: 7.9.2 2024-06-21T01:52:38.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Git sha 0 2024-06-21T01:52:38.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Compile date 2024-06-19 15:40:36 2024-06-21T01:52:38.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: DB SUMMARY 2024-06-21T01:52:38.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: DB Session ID: KYOH5MWO0BSOAD2SSO4D 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: CURRENT file: CURRENT 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: IDENTITY file: IDENTITY 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 902323 ; 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.error_if_exists: 0 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.create_if_missing: 0 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.paranoid_checks: 1 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.env: 0x55b2da798740 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.fs: PosixFileSystem 2024-06-21T01:52:38.480 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.info_log: 0x55b2db6130c0 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_file_opening_threads: 16 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.statistics: (nil) 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.use_fsync: 0 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_log_file_size: 0 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.keep_log_file_num: 1000 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.recycle_log_file_num: 0 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.allow_fallocate: 1 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.allow_mmap_reads: 0 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.allow_mmap_writes: 0 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.use_direct_reads: 0 2024-06-21T01:52:38.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.create_missing_column_families: 0 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.db_log_dir: 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.wal_dir: 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.advise_random_on_open: 1 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.db_write_buffer_size: 0 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.write_buffer_manager: 0x55b2db622c80 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-21T01:52:38.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.rate_limiter: (nil) 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.wal_recovery_mode: 2 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.enable_thread_tracking: 0 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.enable_pipelined_write: 0 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.unordered_write: 0 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.row_cache: None 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.wal_filter: None 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-21T01:52:38.483 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.allow_ingest_behind: 0 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.two_write_queues: 0 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.manual_wal_flush: 0 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.wal_compression: 0 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.atomic_flush: 0 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.log_readahead_size: 0 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.best_efforts_recovery: 0 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.allow_data_in_errors: 0 2024-06-21T01:52:38.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.db_host_id: __hostname__ 2024-06-21T01:52:38.485 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_background_jobs: 2 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_background_compactions: -1 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_subcompactions: 1 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_total_wal_size: 0 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_open_files: -1 2024-06-21T01:52:38.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bytes_per_sync: 0 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_readahead_size: 0 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_background_flushes: -1 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Compression algorithms supported: 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: kZSTD supported: 0 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: kXpressCompression supported: 0 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: kBZip2Compression supported: 0 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: kLZ4Compression supported: 1 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: kZlibCompression supported: 1 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: kLZ4HCCompression supported: 1 2024-06-21T01:52:38.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: kSnappyCompression supported: 1 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.merge_operator: 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_filter: None 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_filter_factory: None 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.sst_partitioner_factory: None 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b2db612d20) 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout: cache_index_and_filter_blocks: 1 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-21T01:52:38.488 INFO:journalctl@ceph.mon.b.smithi173.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: pin_top_level_index_and_filter: 1 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_type: 0 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: data_block_index_type: 0 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_shortening: 1 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: checksum: 4 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: no_block_cache: 0 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache: 0x55b2db60b1f0 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_name: BinnedLRUCache 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_options: 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: capacity : 536870912 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: num_shard_bits : 4 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: strict_capacity_limit : 0 2024-06-21T01:52:38.489 INFO:journalctl@ceph.mon.b.smithi173.stdout: high_pri_pool_ratio: 0.000 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_compressed: (nil) 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: persistent_cache: (nil) 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_size: 4096 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_size_deviation: 10 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_restart_interval: 16 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_block_restart_interval: 1 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: metadata_block_size: 4096 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: partition_filters: 0 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: use_delta_encoding: 1 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: filter_policy: bloomfilter 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: whole_key_filtering: 1 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: verify_compression: 0 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: read_amp_bytes_per_bit: 0 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: format_version: 5 2024-06-21T01:52:38.490 INFO:journalctl@ceph.mon.b.smithi173.stdout: enable_index_compression: 1 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_align: 0 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout: max_auto_readahead_size: 262144 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout: prepopulate_block_cache: 0 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout: initial_auto_readahead_size: 8192 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout: num_file_reads_for_auto_readahead: 2 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.write_buffer_size: 33554432 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_write_buffer_number: 2 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compression: NoCompression 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bottommost_compression: Disabled 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.prefix_extractor: nullptr 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.num_levels: 7 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-21T01:52:38.491 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compression_opts.level: 32767 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compression_opts.strategy: 0 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-21T01:52:38.492 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compression_opts.enabled: false 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.target_file_size_base: 67108864 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-21T01:52:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.arena_block_size: 1048576 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.disable_auto_compactions: 0 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-21T01:52:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.table_properties_collectors: 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.inplace_update_support: 0 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.bloom_locality: 0 2024-06-21T01:52:38.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.max_successive_merges: 0 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.paranoid_file_checks: 0 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.force_consistency_checks: 1 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.report_bg_io_stats: 0 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.ttl: 2592000 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.enable_blob_files: false 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.min_blob_size: 0 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.blob_file_size: 268435456 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-21T01:52:38.496 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.blob_file_starting_level: 0 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14450, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2e1dccc1-7d79-4346-8abe-89dc88dbc9a5 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934758109956, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934758113821, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 538711, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14455, "largest_seqno": 14840, "table_properties": {"data_size": 536075, "index_size": 1110, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4589, "raw_average_key_size": 24, "raw_value_size": 532026, "raw_average_value_size": 2891, "num_data_blocks": 49, "num_entries": 184, "num_filter_entries": 184, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1718934758, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2e1dccc1-7d79-4346-8abe-89dc88dbc9a5", "db_session_id": "KYOH5MWO0BSOAD2SSO4D", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718934758113948, "job": 1, "event": "recovery_finished"} 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-06-21T01:52:38.497 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b2db636e00 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: DB pointer 0x55b2db72c000 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** DB Stats ** 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-21T01:52:38.498 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** Compaction Stats [default] ** 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: L0 1/0 526.08 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 236.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: L6 1/0 10.66 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: Sum 2/0 11.18 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 236.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 236.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** Compaction Stats [default] ** 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 236.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-21T01:52:38.499 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative compaction: 0.00 GB write, 60.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval compaction: 0.00 GB write, 60.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: Block cache BinnedLRUCache@0x55b2db60b1f0#2 capacity: 512.00 MB usage: 1.80 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,1.20 KB,0.000229478%) Misc(1,0.00 KB,0%) 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-21T01:52:38.500 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: starting mon.b rank 2 at public addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] at bind addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: mon.b@-1(???) e3 preinit fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: mon.b@-1(???).mds e1 new map 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: mon.b@-1(???).mds e1 print_map 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout: e1 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout: legacy client fscid: -1 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout: No filesystems configured 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: mon.b@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-mon[57540]: mon.b@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-06-21T01:52:38.501 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 bash[57521]: e6b535ec7a5546c1152bd41779ca2ac70e3b843a76e95b570d26167fedb362ca 2024-06-21T01:52:38.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:38 smithi173 systemd[1]: Started Ceph mon.b for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:52:38.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:52:38 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:52:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:52:39.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:52:39] "GET /metrics HTTP/1.1" 200 38188 "" "Prometheus/2.33.4" 2024-06-21T01:52:43.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:52:43.127+0000 7f9965fe7640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-06-21T01:52:43.854 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:52:43 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:52:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:52:44.196 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:52:44.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: mon.c calling monitor election 2024-06-21T01:52:44.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: mon.b calling monitor election 2024-06-21T01:52:44.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:44.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: last_changed 2024-06-21T01:52:38.248357+0000 2024-06-21T01:52:44.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:44.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: min_mon_release 19 (squid) 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: election_strategy: 1 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: fsmap 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: mgrmap e39: y(active, since 32s), standbys: x 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: overall HEALTH_OK 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: mon.c calling monitor election 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: mon.a calling monitor election 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: monmap epoch 4 2024-06-21T01:52:44.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: last_changed 2024-06-21T01:52:38.248357+0000 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: min_mon_release 19 (squid) 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: election_strategy: 1 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: fsmap 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: mgrmap e39: y(active, since 32s), standbys: x 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: overall HEALTH_OK 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:44.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: mon.c calling monitor election 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: mon.b calling monitor election 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: last_changed 2024-06-21T01:52:38.248357+0000 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: min_mon_release 19 (squid) 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: election_strategy: 1 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: fsmap 2024-06-21T01:52:44.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: mgrmap e39: y(active, since 32s), standbys: x 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: overall HEALTH_OK 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: mon.c calling monitor election 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: mon.a calling monitor election 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: monmap epoch 4 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: last_changed 2024-06-21T01:52:38.248357+0000 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: min_mon_release 19 (squid) 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: election_strategy: 1 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:44.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: fsmap 2024-06-21T01:52:44.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:44.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: mgrmap e39: y(active, since 32s), standbys: x 2024-06-21T01:52:44.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: overall HEALTH_OK 2024-06-21T01:52:44.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:44.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:44.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:44 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:44.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: mon.c calling monitor election 2024-06-21T01:52:44.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: mon.b calling monitor election 2024-06-21T01:52:44.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:44.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: last_changed 2024-06-21T01:52:38.248357+0000 2024-06-21T01:52:44.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:44.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: min_mon_release 19 (squid) 2024-06-21T01:52:44.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: election_strategy: 1 2024-06-21T01:52:44.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:44.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:44.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:44.607 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: fsmap 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: mgrmap e39: y(active, since 32s), standbys: x 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: overall HEALTH_OK 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: mon.c calling monitor election 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: mon.a calling monitor election 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: monmap epoch 4 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: last_changed 2024-06-21T01:52:38.248357+0000 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: created 2024-06-21T01:40:13.962891+0000 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: min_mon_release 19 (squid) 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: election_strategy: 1 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: 0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: 1: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2024-06-21T01:52:44.608 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-06-21T01:52:44.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: fsmap 2024-06-21T01:52:44.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: osdmap e106: 8 total, 8 up, 8 in 2024-06-21T01:52:44.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: mgrmap e39: y(active, since 32s), standbys: x 2024-06-21T01:52:44.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: overall HEALTH_OK 2024-06-21T01:52:44.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:44.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:44.609 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:44 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:44.846 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:52:44.846 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (103s) 15s ago 6m 14.7M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:52:44.846 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (90s) 29s ago 5m 46.9M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:52:44.846 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (61s) 15s ago 5m 48.9M - 3.5 e1d6a67b021e 3192bc59456b 2024-06-21T01:52:44.846 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (54s) 29s ago 10m 436M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:52:44.846 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (84s) 15s ago 12m 487M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (40s) 15s ago 12m 39.3M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 starting - - - 2048M 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (20s) 15s ago 11m 18.8M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (108s) 15s ago 6m 11.2M - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (94s) 29s ago 6m 17.4M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (10m) 15s ago 10m 52.5M 2736M 17.2.0 e1d6a67b021e 63507222842b 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (9m) 15s ago 9m 55.6M 2736M 17.2.0 e1d6a67b021e e0bf7c0600e5 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (9m) 15s ago 9m 48.4M 2736M 17.2.0 e1d6a67b021e 157fb16866c9 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (9m) 15s ago 9m 52.1M 2736M 17.2.0 e1d6a67b021e e5567e44eaec 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (8m) 29s ago 8m 55.5M 2992M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (8m) 29s ago 8m 53.4M 2992M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (8m) 29s ago 8m 48.6M 2992M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (7m) 29s ago 7m 52.7M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:52:44.847 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (59s) 29s ago 7m 59.8M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:52:44.848 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (5m) 15s ago 5m 76.0M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:52:44.848 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (5m) 29s ago 5m 72.2M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:52:44.848 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (5m) 15s ago 5m 73.2M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:52:44.848 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (5m) 29s ago 5m 73.5M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:52:45.238 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:52:45.239 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-06-21T01:52:45.240 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 5 2024-06-21T01:52:45.240 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:52:45.240 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:52:45.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:45 smithi092 ceph-mon[64276]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:45.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:45 smithi092 ceph-mon[64276]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:45.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:45 smithi092 ceph-mon[64276]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:45.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:45 smithi092 ceph-mon[64276]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:45.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:45 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/531839311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:45.570 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:45 smithi092 ceph-mon[66756]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:45.570 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:45 smithi092 ceph-mon[66756]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:45.570 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:45 smithi092 ceph-mon[66756]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:45.570 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:45 smithi092 ceph-mon[66756]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:45.570 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:45 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/531839311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:45.570 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:52:45.570 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:52:45.570 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:52:45.570 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:52:45.570 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:52:45.571 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr" 2024-06-21T01:52:45.571 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:52:45.571 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "4/23 daemons upgraded", 2024-06-21T01:52:45.571 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading mon daemons", 2024-06-21T01:52:45.571 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:52:45.571 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:52:45.573 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:45 smithi173 ceph-mon[57540]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:45.573 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:45 smithi173 ceph-mon[57540]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:45.573 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:45 smithi173 ceph-mon[57540]: from='client.34185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:45.573 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:45 smithi173 ceph-mon[57540]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:45.573 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:45 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/531839311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:52:45.958 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:52:46.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:46 smithi173 ceph-mon[57540]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:46.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:46 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/2405023540' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:52:46.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:46 smithi092 ceph-mon[64276]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:46.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:46 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/2405023540' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:52:46.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:46 smithi092 ceph-mon[66756]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:52:46.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:46 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/2405023540' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:52:47.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:47 smithi092 ceph-mon[66756]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:47.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:47 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:47.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:47 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:47.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:47 smithi092 ceph-mon[64276]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:47.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:47 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:47.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:47 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:47.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:47 smithi173 ceph-mon[57540]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:47.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:47 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:47.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:47 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:48.728 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:52:48 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:52:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:52:49.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:48 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:49.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:48 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:49.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:48 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:49.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:48 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:49.194 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:48 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:49.194 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:48 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:49.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:49 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:52:49] "GET /metrics HTTP/1.1" 200 38261 "" "Prometheus/2.33.4" 2024-06-21T01:52:50.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:49 smithi092 ceph-mon[64276]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:52:50.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:49 smithi092 ceph-mon[66756]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:52:50.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:49 smithi173 ceph-mon[57540]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:52:52.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[64276]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:52.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:52.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[66756]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:52.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:52.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:52.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:52.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:51 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:52.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:51 smithi173 ceph-mon[57540]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:52.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:51 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:52.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:51 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:52.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:51 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:52.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:51 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:52:52.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:51 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:52.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:51 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:52.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:51 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:52.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:51 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:53.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:52 smithi173 ceph-mon[57540]: Reconfiguring mon.a (monmap changed)... 2024-06-21T01:52:53.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:52 smithi173 ceph-mon[57540]: Reconfiguring daemon mon.a on smithi092 2024-06-21T01:52:53.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:52 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:53.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:52 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:53.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:52 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:52:53.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:52 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:52:53.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:52 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:53.239 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[64276]: Reconfiguring mon.a (monmap changed)... 2024-06-21T01:52:53.239 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[64276]: Reconfiguring daemon mon.a on smithi092 2024-06-21T01:52:53.239 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:53.239 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:53.239 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:52:53.240 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:52:53.240 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:53.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[66756]: Reconfiguring mon.a (monmap changed)... 2024-06-21T01:52:53.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[66756]: Reconfiguring daemon mon.a on smithi092 2024-06-21T01:52:53.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:53.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:53.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:52:53.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:52:53.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:52 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:53.929 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:52:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:52:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:52:54.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:53 smithi173 ceph-mon[57540]: Reconfiguring mgr.y (monmap changed)... 2024-06-21T01:52:54.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:53 smithi173 ceph-mon[57540]: Reconfiguring daemon mgr.y on smithi092 2024-06-21T01:52:54.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:53 smithi173 ceph-mon[57540]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:52:54.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:53 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:54.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:53 smithi092 ceph-mon[64276]: Reconfiguring mgr.y (monmap changed)... 2024-06-21T01:52:54.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:53 smithi092 ceph-mon[64276]: Reconfiguring daemon mgr.y on smithi092 2024-06-21T01:52:54.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:53 smithi092 ceph-mon[64276]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:52:54.350 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:53 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:54.350 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:53 smithi092 ceph-mon[66756]: Reconfiguring mgr.y (monmap changed)... 2024-06-21T01:52:54.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:53 smithi092 ceph-mon[66756]: Reconfiguring daemon mgr.y on smithi092 2024-06-21T01:52:54.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:53 smithi092 ceph-mon[66756]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:52:54.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:53 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:52:55.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:55.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:55.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[64276]: Reconfiguring mon.c (monmap changed)... 2024-06-21T01:52:55.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:55.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:55.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:55.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[64276]: Reconfiguring daemon mon.c on smithi092 2024-06-21T01:52:55.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[64276]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:55.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:55.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:55.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[66756]: Reconfiguring mon.c (monmap changed)... 2024-06-21T01:52:55.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:55.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:55.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:55.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[66756]: Reconfiguring daemon mon.c on smithi092 2024-06-21T01:52:55.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:54 smithi092 ceph-mon[66756]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:55.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:54 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:54 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:54 smithi173 ceph-mon[57540]: Reconfiguring mon.c (monmap changed)... 2024-06-21T01:52:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:54 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:52:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:54 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:52:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:54 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:54 smithi173 ceph-mon[57540]: Reconfiguring daemon mon.c on smithi092 2024-06-21T01:52:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:54 smithi173 ceph-mon[57540]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:56.572 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:56.572 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:56.572 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[64276]: Reconfiguring osd.0 (monmap changed)... 2024-06-21T01:52:56.572 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-21T01:52:56.572 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:56.572 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[64276]: Reconfiguring daemon osd.0 on smithi092 2024-06-21T01:52:56.572 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:56.572 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:56.573 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:56.573 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[66756]: Reconfiguring osd.0 (monmap changed)... 2024-06-21T01:52:56.573 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-21T01:52:56.573 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:56.573 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[66756]: Reconfiguring daemon osd.0 on smithi092 2024-06-21T01:52:56.573 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:56.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:56.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:56.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:56 smithi173 ceph-mon[57540]: Reconfiguring osd.0 (monmap changed)... 2024-06-21T01:52:56.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-21T01:52:56.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:56.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:56 smithi173 ceph-mon[57540]: Reconfiguring daemon osd.0 on smithi092 2024-06-21T01:52:56.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:52:57.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:57 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:57.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:57 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:57.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:57 smithi173 ceph-mon[57540]: Reconfiguring osd.1 (monmap changed)... 2024-06-21T01:52:57.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:57 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-21T01:52:57.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:57 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:57.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:57 smithi173 ceph-mon[57540]: Reconfiguring daemon osd.1 on smithi092 2024-06-21T01:52:57.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:57 smithi173 ceph-mon[57540]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:58.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:58.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:58.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[64276]: Reconfiguring osd.1 (monmap changed)... 2024-06-21T01:52:58.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-21T01:52:58.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:58.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[64276]: Reconfiguring daemon osd.1 on smithi092 2024-06-21T01:52:58.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[64276]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:58.038 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:58.038 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:58.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[66756]: Reconfiguring osd.1 (monmap changed)... 2024-06-21T01:52:58.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-21T01:52:58.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:58.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[66756]: Reconfiguring daemon osd.1 on smithi092 2024-06-21T01:52:58.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:57 smithi092 ceph-mon[66756]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:52:58.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:52:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:52:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:52:59.402 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:52:59] "GET /metrics HTTP/1.1" 200 38263 "" "Prometheus/2.33.4" 2024-06-21T01:52:59.402 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:59.402 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:59.402 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[64276]: Reconfiguring osd.2 (monmap changed)... 2024-06-21T01:52:59.402 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-21T01:52:59.402 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:59.403 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[64276]: Reconfiguring daemon osd.2 on smithi092 2024-06-21T01:52:59.403 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[64276]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:52:59.403 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:59.403 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:59.403 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[66756]: Reconfiguring osd.2 (monmap changed)... 2024-06-21T01:52:59.403 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-21T01:52:59.403 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:59.403 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[66756]: Reconfiguring daemon osd.2 on smithi092 2024-06-21T01:52:59.403 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:52:59 smithi092 ceph-mon[66756]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:52:59.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:59 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:59.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:59 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:52:59.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:59 smithi173 ceph-mon[57540]: Reconfiguring osd.2 (monmap changed)... 2024-06-21T01:52:59.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:59 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-21T01:52:59.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:59 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:52:59.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:59 smithi173 ceph-mon[57540]: Reconfiguring daemon osd.2 on smithi092 2024-06-21T01:52:59.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:52:59 smithi173 ceph-mon[57540]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:00.802 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:00.802 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:00.802 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[64276]: Reconfiguring osd.3 (monmap changed)... 2024-06-21T01:53:00.802 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-21T01:53:00.802 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:00.802 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[64276]: Reconfiguring daemon osd.3 on smithi092 2024-06-21T01:53:00.802 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:00.802 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:00.803 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[66756]: Reconfiguring osd.3 (monmap changed)... 2024-06-21T01:53:00.803 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-21T01:53:00.803 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:00.803 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:00 smithi092 ceph-mon[66756]: Reconfiguring daemon osd.3 on smithi092 2024-06-21T01:53:00.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:00 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:00.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:00 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:00.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:00 smithi173 ceph-mon[57540]: Reconfiguring osd.3 (monmap changed)... 2024-06-21T01:53:00.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:00 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-21T01:53:00.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:00 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:00.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:00 smithi173 ceph-mon[57540]: Reconfiguring daemon osd.3 on smithi092 2024-06-21T01:53:02.184 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[66756]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:02.184 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:02.184 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:02.184 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[66756]: Reconfiguring rgw.foo.smithi092.lzbkvh (monmap changed)... 2024-06-21T01:53:02.185 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:02.185 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:02.185 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[66756]: Reconfiguring daemon rgw.foo.smithi092.lzbkvh on smithi092 2024-06-21T01:53:02.185 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[64276]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:02.185 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:02.185 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:02.185 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[64276]: Reconfiguring rgw.foo.smithi092.lzbkvh (monmap changed)... 2024-06-21T01:53:02.185 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:02.185 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:02.185 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:01 smithi092 ceph-mon[64276]: Reconfiguring daemon rgw.foo.smithi092.lzbkvh on smithi092 2024-06-21T01:53:02.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:01 smithi173 ceph-mon[57540]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:02.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:01 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:02.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:01 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:02.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:01 smithi173 ceph-mon[57540]: Reconfiguring rgw.foo.smithi092.lzbkvh (monmap changed)... 2024-06-21T01:53:02.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:01 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:02.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:01 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:02.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:01 smithi173 ceph-mon[57540]: Reconfiguring daemon rgw.foo.smithi092.lzbkvh on smithi092 2024-06-21T01:53:03.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:03 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:03.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:03 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:03.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:03 smithi173 ceph-mon[57540]: Reconfiguring rgw.smpl.smithi092.bxxcjp (monmap changed)... 2024-06-21T01:53:03.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:03 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:03.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:03 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:03.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:03 smithi173 ceph-mon[57540]: Reconfiguring daemon rgw.smpl.smithi092.bxxcjp on smithi092 2024-06-21T01:53:03.502 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:03 smithi173 ceph-mon[57540]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:03.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:03.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:03.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[64276]: Reconfiguring rgw.smpl.smithi092.bxxcjp (monmap changed)... 2024-06-21T01:53:03.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:03.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:03.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[64276]: Reconfiguring daemon rgw.smpl.smithi092.bxxcjp on smithi092 2024-06-21T01:53:03.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[64276]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:03.532 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:03.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:03.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[66756]: Reconfiguring rgw.smpl.smithi092.bxxcjp (monmap changed)... 2024-06-21T01:53:03.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:03.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:03.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[66756]: Reconfiguring daemon rgw.smpl.smithi092.bxxcjp on smithi092 2024-06-21T01:53:03.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:03 smithi092 ceph-mon[66756]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:03.962 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:04.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:04.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:04.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[64276]: Reconfiguring mon.b (monmap changed)... 2024-06-21T01:53:04.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:53:04.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:53:04.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:04.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[64276]: Reconfiguring daemon mon.b on smithi173 2024-06-21T01:53:04.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:04.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:04.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:04.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[66756]: Reconfiguring mon.b (monmap changed)... 2024-06-21T01:53:04.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:53:04.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:53:04.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:04.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[66756]: Reconfiguring daemon mon.b on smithi173 2024-06-21T01:53:04.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:04 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:04.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:04 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:04.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:04 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:04.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:04 smithi173 ceph-mon[57540]: Reconfiguring mon.b (monmap changed)... 2024-06-21T01:53:04.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:04 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-21T01:53:04.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:04 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-21T01:53:04.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:04 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:04.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:04 smithi173 ceph-mon[57540]: Reconfiguring daemon mon.b on smithi173 2024-06-21T01:53:04.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:04 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:05.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[64276]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:05.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:05.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:05.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:53:05.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:53:05.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:05.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[66756]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:05.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:05.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:05.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:53:05.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:53:05.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:05 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:05 smithi173 ceph-mon[57540]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:05 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:05 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:05 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-21T01:53:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:05 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-21T01:53:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:05 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:06.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:06 smithi173 ceph-mon[57540]: Reconfiguring mgr.x (monmap changed)... 2024-06-21T01:53:06.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:06 smithi173 ceph-mon[57540]: Reconfiguring daemon mgr.x on smithi173 2024-06-21T01:53:06.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:06.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:06.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-21T01:53:06.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:06.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[64276]: Reconfiguring mgr.x (monmap changed)... 2024-06-21T01:53:06.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[64276]: Reconfiguring daemon mgr.x on smithi173 2024-06-21T01:53:06.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:06.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:06.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-21T01:53:06.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[66756]: Reconfiguring mgr.x (monmap changed)... 2024-06-21T01:53:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[66756]: Reconfiguring daemon mgr.x on smithi173 2024-06-21T01:53:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-21T01:53:06.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:07.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:07 smithi173 ceph-mon[57540]: Reconfiguring osd.4 (monmap changed)... 2024-06-21T01:53:07.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:07 smithi173 ceph-mon[57540]: Reconfiguring daemon osd.4 on smithi173 2024-06-21T01:53:07.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:07 smithi173 ceph-mon[57540]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:07.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:07 smithi092 ceph-mon[64276]: Reconfiguring osd.4 (monmap changed)... 2024-06-21T01:53:07.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:07 smithi092 ceph-mon[64276]: Reconfiguring daemon osd.4 on smithi173 2024-06-21T01:53:07.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:07 smithi092 ceph-mon[64276]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:07.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:07 smithi092 ceph-mon[66756]: Reconfiguring osd.4 (monmap changed)... 2024-06-21T01:53:07.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:07 smithi092 ceph-mon[66756]: Reconfiguring daemon osd.4 on smithi173 2024-06-21T01:53:07.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:07 smithi092 ceph-mon[66756]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:08.817 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:09.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:08 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:09.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:08 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:09.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:08 smithi173 ceph-mon[57540]: Reconfiguring osd.5 (monmap changed)... 2024-06-21T01:53:09.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:08 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-21T01:53:09.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:08 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:09.093 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:08 smithi173 ceph-mon[57540]: Reconfiguring daemon osd.5 on smithi173 2024-06-21T01:53:09.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:09.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:09.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[64276]: Reconfiguring osd.5 (monmap changed)... 2024-06-21T01:53:09.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-21T01:53:09.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:09.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[64276]: Reconfiguring daemon osd.5 on smithi173 2024-06-21T01:53:09.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:09.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:09.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[66756]: Reconfiguring osd.5 (monmap changed)... 2024-06-21T01:53:09.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-21T01:53:09.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:09.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:08 smithi092 ceph-mon[66756]: Reconfiguring daemon osd.5 on smithi173 2024-06-21T01:53:09.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:53:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:53:09] "GET /metrics HTTP/1.1" 200 38263 "" "Prometheus/2.33.4" 2024-06-21T01:53:10.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:09 smithi173 ceph-mon[57540]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:10.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:09 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:10.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:09 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:10.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:09 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-21T01:53:10.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:09 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:10.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:09 smithi092 ceph-mon[64276]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:10.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:09 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:10.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:09 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:10.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:09 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-21T01:53:10.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:09 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:10.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:09 smithi092 ceph-mon[66756]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:10.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:09 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:10.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:09 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:10.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:09 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-21T01:53:10.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:09 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:11.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:10 smithi173 ceph-mon[57540]: Reconfiguring osd.6 (monmap changed)... 2024-06-21T01:53:11.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:10 smithi173 ceph-mon[57540]: Reconfiguring daemon osd.6 on smithi173 2024-06-21T01:53:11.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:10 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:11.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:10 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:11.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:10 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-21T01:53:11.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:10 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:11.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[64276]: Reconfiguring osd.6 (monmap changed)... 2024-06-21T01:53:11.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[64276]: Reconfiguring daemon osd.6 on smithi173 2024-06-21T01:53:11.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:11.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:11.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-21T01:53:11.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:11.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[66756]: Reconfiguring osd.6 (monmap changed)... 2024-06-21T01:53:11.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[66756]: Reconfiguring daemon osd.6 on smithi173 2024-06-21T01:53:11.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:11.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:11.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-21T01:53:11.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:10 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:11 smithi173 ceph-mon[57540]: Reconfiguring osd.7 (monmap changed)... 2024-06-21T01:53:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:11 smithi173 ceph-mon[57540]: Reconfiguring daemon osd.7 on smithi173 2024-06-21T01:53:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:11 smithi173 ceph-mon[57540]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:11 smithi092 ceph-mon[64276]: Reconfiguring osd.7 (monmap changed)... 2024-06-21T01:53:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:11 smithi092 ceph-mon[64276]: Reconfiguring daemon osd.7 on smithi173 2024-06-21T01:53:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:11 smithi092 ceph-mon[64276]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:12.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:11 smithi092 ceph-mon[66756]: Reconfiguring osd.7 (monmap changed)... 2024-06-21T01:53:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:11 smithi092 ceph-mon[66756]: Reconfiguring daemon osd.7 on smithi173 2024-06-21T01:53:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:11 smithi092 ceph-mon[66756]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:13.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:12 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:13.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:12 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:13.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:12 smithi173 ceph-mon[57540]: Reconfiguring rgw.foo.smithi173.hvgnvt (monmap changed)... 2024-06-21T01:53:13.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:12 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:13.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:12 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:13.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:12 smithi173 ceph-mon[57540]: Reconfiguring daemon rgw.foo.smithi173.hvgnvt on smithi173 2024-06-21T01:53:13.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:12 smithi173 ceph-mon[57540]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[64276]: Reconfiguring rgw.foo.smithi173.hvgnvt (monmap changed)... 2024-06-21T01:53:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:13.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:13.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[64276]: Reconfiguring daemon rgw.foo.smithi173.hvgnvt on smithi173 2024-06-21T01:53:13.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[64276]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:13.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:13.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:13.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[66756]: Reconfiguring rgw.foo.smithi173.hvgnvt (monmap changed)... 2024-06-21T01:53:13.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:13.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:13.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[66756]: Reconfiguring daemon rgw.foo.smithi173.hvgnvt on smithi173 2024-06-21T01:53:13.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:12 smithi092 ceph-mon[66756]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:13.759 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:14.599 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:14 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:14.599 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:14 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:14.599 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:14 smithi173 ceph-mon[57540]: Reconfiguring rgw.smpl.smithi173.khwpnd (monmap changed)... 2024-06-21T01:53:14.599 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:14 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:14.599 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:14 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:14.599 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:14 smithi173 ceph-mon[57540]: Reconfiguring daemon rgw.smpl.smithi173.khwpnd on smithi173 2024-06-21T01:53:14.599 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:14 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:14.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:14.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:14.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[64276]: Reconfiguring rgw.smpl.smithi173.khwpnd (monmap changed)... 2024-06-21T01:53:14.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:14.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:14.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[64276]: Reconfiguring daemon rgw.smpl.smithi173.khwpnd on smithi173 2024-06-21T01:53:14.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:14.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:14.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:14.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[66756]: Reconfiguring rgw.smpl.smithi173.khwpnd (monmap changed)... 2024-06-21T01:53:14.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:53:14.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:14.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[66756]: Reconfiguring daemon rgw.smpl.smithi173.khwpnd on smithi173 2024-06-21T01:53:14.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:14 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:15.937 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:53:15.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:15.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:15.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: Upgrade: Setting container_image for all mon 2024-06-21T01:53:15.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-06-21T01:53:15.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-06-21T01:53:15.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-06-21T01:53:15.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-06-21T01:53:15.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-06-21T01:53:15.939 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-06-21T01:53:15.939 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:15.939 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: Upgrade: Setting container_image for all crash 2024-06-21T01:53:15.939 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.939 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-21T01:53:15.939 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-21T01:53:15.939 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: Upgrade: osd.0 is safe to restart 2024-06-21T01:53:15.939 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[64276]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:15.939 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.939 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.939 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: Upgrade: Setting container_image for all mon 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: Upgrade: Setting container_image for all crash 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.940 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-21T01:53:15.941 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-21T01:53:15.941 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: Upgrade: osd.0 is safe to restart 2024-06-21T01:53:15.941 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:15 smithi092 ceph-mon[66756]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:15.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: Upgrade: Setting container_image for all mon 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-06-21T01:53:15.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:15.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: Upgrade: Setting container_image for all crash 2024-06-21T01:53:15.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:15.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-21T01:53:15.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-21T01:53:15.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: Upgrade: osd.0 is safe to restart 2024-06-21T01:53:15.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:15 smithi173 ceph-mon[57540]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:16.356 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:53:17.027 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:53:17.027 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (2m) 47s ago 6m 14.7M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:53:17.027 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (2m) 29s ago 6m 46.8M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:53:17.027 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (94s) 47s ago 5m 48.9M - 3.5 e1d6a67b021e 3192bc59456b 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (86s) 29s ago 11m 438M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (116s) 47s ago 12m 487M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (73s) 47s ago 13m 39.3M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (39s) 29s ago 11m 22.2M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (52s) 47s ago 11m 18.8M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (2m) 47s ago 7m 11.2M - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (2m) 29s ago 7m 20.3M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (10m) 47s ago 10m 52.5M 2736M 17.2.0 e1d6a67b021e 63507222842b 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (10m) 47s ago 10m 55.6M 2736M 17.2.0 e1d6a67b021e e0bf7c0600e5 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (10m) 47s ago 10m 48.4M 2736M 17.2.0 e1d6a67b021e 157fb16866c9 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (9m) 47s ago 9m 52.1M 2736M 17.2.0 e1d6a67b021e e5567e44eaec 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (9m) 29s ago 9m 56.0M 2992M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (9m) 29s ago 9m 54.0M 2992M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:53:17.028 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (8m) 29s ago 8m 48.9M 2992M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:53:17.029 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (8m) 29s ago 8m 53.2M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:53:17.029 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (91s) 29s ago 7m 60.0M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:53:17.029 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (5m) 47s ago 5m 76.0M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:53:17.029 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (5m) 29s ago 5m 72.6M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:53:17.029 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (5m) 47s ago 5m 73.2M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:53:17.029 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (5m) 29s ago 5m 73.8M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:53:17.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:16 smithi173 ceph-mon[57540]: Upgrade: Updating osd.0 2024-06-21T01:53:17.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:16 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:17.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:16 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-21T01:53:17.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:16 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:17.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:16 smithi173 ceph-mon[57540]: Deploying daemon osd.0 on smithi092 2024-06-21T01:53:17.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:16 smithi173 ceph-mon[57540]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:17.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:16 smithi173 ceph-mon[57540]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:17.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:16 smithi173 ceph-mon[57540]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[64276]: Upgrade: Updating osd.0 2024-06-21T01:53:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-21T01:53:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[64276]: Deploying daemon osd.0 on smithi092 2024-06-21T01:53:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[64276]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:17.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[64276]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:17.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[64276]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[66756]: Upgrade: Updating osd.0 2024-06-21T01:53:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-21T01:53:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[66756]: Deploying daemon osd.0 on smithi092 2024-06-21T01:53:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[66756]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[66756]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:16 smithi092 ceph-mon[66756]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:17.464 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:53:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:53:17.466 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:53:17.466 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-06-21T01:53:17.466 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 5 2024-06-21T01:53:17.466 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:53:17.466 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:53:17.814 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:53:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:53:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:53:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:53:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:53:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr", 2024-06-21T01:53:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "mon" 2024-06-21T01:53:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:53:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "5/23 daemons upgraded", 2024-06-21T01:53:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading osd daemons", 2024-06-21T01:53:17.815 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:53:17.815 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:53:18.228 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:53:18.228 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:17 smithi092 ceph-mon[64276]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:18.228 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:17 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/971330690' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:18.228 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:17 smithi092 ceph-mon[64276]: from='client.44161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:18.229 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:17 smithi092 ceph-mon[66756]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:18.229 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:17 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/971330690' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:18.229 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:17 smithi092 ceph-mon[66756]: from='client.44161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:18.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:17 smithi173 ceph-mon[57540]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:18.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:17 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/971330690' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:18.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:17 smithi173 ceph-mon[57540]: from='client.44161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:18.851 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:18 smithi092 systemd[1]: Stopping Ceph osd.0 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:53:18.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:19.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:18 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/1246622466' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:53:19.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:18 smithi092 ceph-mon[64276]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:19.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:18 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/1246622466' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:53:19.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:18 smithi092 ceph-mon[66756]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:19.266 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[32647]: 2024-06-21T01:53:19.033+0000 7f10483bd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:53:19.267 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[32647]: 2024-06-21T01:53:19.033+0000 7f10483bd700 -1 osd.0 106 *** Got signal Terminated *** 2024-06-21T01:53:19.267 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[32647]: 2024-06-21T01:53:19.033+0000 7f10483bd700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T01:53:19.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:18 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/1246622466' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:53:19.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:18 smithi173 ceph-mon[57540]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:19.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:53:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:53:19] "GET /metrics HTTP/1.1" 200 38258 "" "Prometheus/2.33.4" 2024-06-21T01:53:20.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:19 smithi092 ceph-mon[66756]: osd.0 marked itself down and dead 2024-06-21T01:53:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:19 smithi092 ceph-mon[64276]: osd.0 marked itself down and dead 2024-06-21T01:53:20.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:19 smithi173 ceph-mon[57540]: osd.0 marked itself down and dead 2024-06-21T01:53:20.713 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:20 smithi092 podman[69635]: 2024-06-21 01:53:20.448407394 +0000 UTC m=+1.525224238 container died 63507222842b9cb578ccae0216256bf3b4aa604c95dfb2e366a5d243009a23b6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, vcs-type=git, maintainer=Guillaume Abrioux ) 2024-06-21T01:53:20.984 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:20 smithi092 podman[69635]: 2024-06-21 01:53:20.808099572 +0000 UTC m=+1.884916444 container cleanup 63507222842b9cb578ccae0216256bf3b4aa604c95dfb2e366a5d243009a23b6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, ceph=True, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, name=centos-stream) 2024-06-21T01:53:20.985 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:20 smithi092 bash[69635]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0 2024-06-21T01:53:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:20 smithi092 ceph-mon[64276]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:53:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:20 smithi092 ceph-mon[64276]: osdmap e107: 8 total, 7 up, 8 in 2024-06-21T01:53:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:20 smithi092 ceph-mon[64276]: pgmap v39: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:53:21.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:20 smithi092 ceph-mon[66756]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:53:21.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:20 smithi092 ceph-mon[66756]: osdmap e107: 8 total, 7 up, 8 in 2024-06-21T01:53:21.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:20 smithi092 ceph-mon[66756]: pgmap v39: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:53:21.352 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:21 smithi092 podman[69661]: 2024-06-21 01:53:21.067849121 +0000 UTC m=+0.622028703 container remove 63507222842b9cb578ccae0216256bf3b4aa604c95dfb2e366a5d243009a23b6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, release=754, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-06-21T01:53:21.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:20 smithi173 ceph-mon[57540]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:53:21.380 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:20 smithi173 ceph-mon[57540]: osdmap e107: 8 total, 7 up, 8 in 2024-06-21T01:53:21.380 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:20 smithi173 ceph-mon[57540]: pgmap v39: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:53:21.684 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:21 smithi092 podman[69753]: 2024-06-21 01:53:21.331339366 +0000 UTC m=+0.017291941 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:53:21.684 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:21 smithi092 podman[69753]: 2024-06-21 01:53:21.443746809 +0000 UTC m=+0.129699427 container create 5efe42cf48c34cae3b3d25644bc209da2916cc8c1ded48a60e36e2f21147eb44 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-21T01:53:21.953 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:21 smithi092 podman[69753]: 2024-06-21 01:53:21.687956846 +0000 UTC m=+0.373909421 container init 5efe42cf48c34cae3b3d25644bc209da2916cc8c1ded48a60e36e2f21147eb44 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, ceph=True, RELEASE=squid-release-c65be81, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T01:53:21.953 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:21 smithi092 podman[69753]: 2024-06-21 01:53:21.69160895 +0000 UTC m=+0.377561526 container start 5efe42cf48c34cae3b3d25644bc209da2916cc8c1ded48a60e36e2f21147eb44 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-release-c65be81, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-21T01:53:21.953 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:21 smithi092 podman[69753]: 2024-06-21 01:53:21.787892186 +0000 UTC m=+0.473844770 container attach 5efe42cf48c34cae3b3d25644bc209da2916cc8c1ded48a60e36e2f21147eb44 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81) 2024-06-21T01:53:22.247 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:21 smithi092 ceph-mon[64276]: osdmap e108: 8 total, 7 up, 8 in 2024-06-21T01:53:22.247 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:21 smithi092 ceph-mon[66756]: osdmap e108: 8 total, 7 up, 8 in 2024-06-21T01:53:22.247 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:21 smithi092 podman[69753]: 2024-06-21 01:53:21.956867323 +0000 UTC m=+0.642819899 container died 5efe42cf48c34cae3b3d25644bc209da2916cc8c1ded48a60e36e2f21147eb44 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:53:22.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:21 smithi173 ceph-mon[57540]: osdmap e108: 8 total, 7 up, 8 in 2024-06-21T01:53:22.851 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:22 smithi092 podman[69776]: 2024-06-21 01:53:22.549795948 +0000 UTC m=+0.586959517 container remove 5efe42cf48c34cae3b3d25644bc209da2916cc8c1ded48a60e36e2f21147eb44 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, GIT_CLEAN=True) 2024-06-21T01:53:22.852 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:22 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.0.service: Deactivated successfully. 2024-06-21T01:53:22.852 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:22 smithi092 systemd[1]: Stopped Ceph osd.0 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:53:22.852 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:22 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.0.service: Consumed 6.846s CPU time. 2024-06-21T01:53:22.852 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:22 smithi092 systemd[1]: Starting Ceph osd.0 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:53:23.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:23 smithi092 ceph-mon[64276]: pgmap v41: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:23.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:23 smithi092 ceph-mon[66756]: pgmap v41: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:23.266 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 podman[69876]: 2024-06-21 01:53:22.915408358 +0000 UTC m=+0.017731320 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:53:23.267 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 podman[69876]: 2024-06-21 01:53:23.05081034 +0000 UTC m=+0.153133332 container create ca70865d4312eeebcdcc7cfc50d243bbfec42a748ee61ea640c0a3ee3ff5fd63 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS) 2024-06-21T01:53:23.267 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 podman[69876]: 2024-06-21 01:53:23.270177949 +0000 UTC m=+0.372500918 container init ca70865d4312eeebcdcc7cfc50d243bbfec42a748ee61ea640c0a3ee3ff5fd63 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, ceph=True, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-06-21T01:53:23.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:23 smithi173 ceph-mon[57540]: pgmap v41: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:23.601 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 podman[69876]: 2024-06-21 01:53:23.273553701 +0000 UTC m=+0.375876651 container start ca70865d4312eeebcdcc7cfc50d243bbfec42a748ee61ea640c0a3ee3ff5fd63 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618) 2024-06-21T01:53:23.601 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 podman[69876]: 2024-06-21 01:53:23.336426223 +0000 UTC m=+0.438749174 container attach ca70865d4312eeebcdcc7cfc50d243bbfec42a748ee61ea640c0a3ee3ff5fd63 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-06-21T01:53:23.977 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate[69891]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-21T01:53:23.977 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 bash[69876]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-21T01:53:23.977 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate[69891]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-21T01:53:23.977 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 bash[69876]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-21T01:53:23.977 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate[69891]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-21T01:53:23.977 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 bash[69876]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-21T01:53:23.977 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate[69891]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-21T01:53:23.978 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 bash[69876]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-21T01:53:23.978 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate[69891]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-06-21T01:53:23.978 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 bash[69876]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-06-21T01:53:23.978 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate[69891]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-21T01:53:23.978 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 bash[69876]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-21T01:53:23.978 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate[69891]: --> ceph-volume raw activate successful for osd ID: 0 2024-06-21T01:53:23.978 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 bash[69876]: --> ceph-volume raw activate successful for osd ID: 0 2024-06-21T01:53:23.978 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:23 smithi092 podman[69876]: 2024-06-21 01:53:23.690379932 +0000 UTC m=+0.792702888 container died ca70865d4312eeebcdcc7cfc50d243bbfec42a748ee61ea640c0a3ee3ff5fd63 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=) 2024-06-21T01:53:23.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:23 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:24.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:24 smithi092 ceph-mon[64276]: Health check failed: Degraded data redundancy: 77/723 objects degraded (10.650%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:53:24.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:24 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:24.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:24 smithi092 ceph-mon[66756]: Health check failed: Degraded data redundancy: 77/723 objects degraded (10.650%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:53:24.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:24 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:24.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:24 smithi173 ceph-mon[57540]: Health check failed: Degraded data redundancy: 77/723 objects degraded (10.650%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:53:24.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:24 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:24.771 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:24 smithi092 podman[70019]: 2024-06-21 01:53:24.406718909 +0000 UTC m=+0.706106303 container remove ca70865d4312eeebcdcc7cfc50d243bbfec42a748ee61ea640c0a3ee3ff5fd63 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-06-21T01:53:24.772 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:24 smithi092 podman[70079]: 2024-06-21 01:53:24.524009995 +0000 UTC m=+0.016909061 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:53:24.772 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:24 smithi092 podman[70079]: 2024-06-21 01:53:24.640997003 +0000 UTC m=+0.133896069 container create 7c252b0f2dfd0299e96f8c1c98c45fbc7beec3091b1c6462878b47b9b447a809 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-06-21T01:53:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:25 smithi092 ceph-mon[64276]: pgmap v42: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:25 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:25.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:25 smithi092 ceph-mon[66756]: pgmap v42: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:25.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:25 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:25.102 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:24 smithi092 podman[70079]: 2024-06-21 01:53:24.868182202 +0000 UTC m=+0.361081268 container init 7c252b0f2dfd0299e96f8c1c98c45fbc7beec3091b1c6462878b47b9b447a809 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-21T01:53:25.102 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:24 smithi092 podman[70079]: 2024-06-21 01:53:24.871303704 +0000 UTC m=+0.364202772 container start 7c252b0f2dfd0299e96f8c1c98c45fbc7beec3091b1c6462878b47b9b447a809 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-21T01:53:25.102 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:24 smithi092 ceph-osd[70098]: -- 172.21.15.92:0/4139925600 <== mon.1 v2:172.21.15.92:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fa24c65a40 con 0x55fa25a24000 2024-06-21T01:53:25.102 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:24 smithi092 bash[70079]: 7c252b0f2dfd0299e96f8c1c98c45fbc7beec3091b1c6462878b47b9b447a809 2024-06-21T01:53:25.102 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:24 smithi092 systemd[1]: Started Ceph osd.0 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:53:25.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:25 smithi173 ceph-mon[57540]: pgmap v42: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:25.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:25 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:25.676 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:25 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[70094]: 2024-06-21T01:53:25.411+0000 7f6a9e7e2640 -1 Falling back to public interface 2024-06-21T01:53:26.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:26.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:26.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:26.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:26.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:26.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:26.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:26.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:26.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:27.320 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:27 smithi092 ceph-mon[64276]: pgmap v43: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:27.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:27 smithi092 ceph-mon[66756]: pgmap v43: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:27.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:27 smithi173 ceph-mon[57540]: pgmap v43: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:28.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:28 smithi092 ceph-mon[64276]: from='osd.0 [v2:172.21.15.92:6802/3740442374,v1:172.21.15.92:6803/3740442374]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:53:28.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:28 smithi092 ceph-mon[64276]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:53:28.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:28 smithi092 ceph-mon[66756]: from='osd.0 [v2:172.21.15.92:6802/3740442374,v1:172.21.15.92:6803/3740442374]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:53:28.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:28 smithi092 ceph-mon[66756]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:53:28.601 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:28 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[70094]: 2024-06-21T01:53:28.154+0000 7f6a9e7e2640 -1 osd.0 0 read_superblock omap replica is missing. 2024-06-21T01:53:28.602 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:28 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[70094]: 2024-06-21T01:53:28.178+0000 7f6a9e7e2640 -1 osd.0 106 log_to_monitors true 2024-06-21T01:53:28.655 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:28 smithi173 ceph-mon[57540]: from='osd.0 [v2:172.21.15.92:6802/3740442374,v1:172.21.15.92:6803/3740442374]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:53:28.656 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:28 smithi173 ceph-mon[57540]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-21T01:53:28.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:28 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:29.101 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 01:53:28 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[70094]: 2024-06-21T01:53:28.684+0000 7f6a92585640 -1 osd.0 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:53:29.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:53:29] "GET /metrics HTTP/1.1" 200 38392 "" "Prometheus/2.33.4" 2024-06-21T01:53:29.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[64276]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-21T01:53:29.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[64276]: osdmap e109: 8 total, 7 up, 8 in 2024-06-21T01:53:29.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[64276]: from='osd.0 [v2:172.21.15.92:6802/3740442374,v1:172.21.15.92:6803/3740442374]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:53:29.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[64276]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:53:29.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[64276]: from='osd.0 ' entity='osd.0' 2024-06-21T01:53:29.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[64276]: pgmap v45: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:29.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:29.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:29.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[66756]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-21T01:53:29.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[66756]: osdmap e109: 8 total, 7 up, 8 in 2024-06-21T01:53:29.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[66756]: from='osd.0 [v2:172.21.15.92:6802/3740442374,v1:172.21.15.92:6803/3740442374]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:53:29.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[66756]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:53:29.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[66756]: from='osd.0 ' entity='osd.0' 2024-06-21T01:53:29.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[66756]: pgmap v45: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:29.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:29.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:29 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:29.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:29 smithi173 ceph-mon[57540]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-21T01:53:29.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:29 smithi173 ceph-mon[57540]: osdmap e109: 8 total, 7 up, 8 in 2024-06-21T01:53:29.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:29 smithi173 ceph-mon[57540]: from='osd.0 [v2:172.21.15.92:6802/3740442374,v1:172.21.15.92:6803/3740442374]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:53:29.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:29 smithi173 ceph-mon[57540]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:53:29.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:29 smithi173 ceph-mon[57540]: from='osd.0 ' entity='osd.0' 2024-06-21T01:53:29.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:29 smithi173 ceph-mon[57540]: pgmap v45: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-21T01:53:29.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:29 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:29.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:29 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:30 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:30 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:30 smithi173 ceph-mon[57540]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:53:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:30 smithi173 ceph-mon[57540]: osd.0 [v2:172.21.15.92:6802/3740442374,v1:172.21.15.92:6803/3740442374] boot 2024-06-21T01:53:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:30 smithi173 ceph-mon[57540]: osdmap e110: 8 total, 8 up, 8 in 2024-06-21T01:53:30.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:30 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:53:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[64276]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:53:31.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[64276]: osd.0 [v2:172.21.15.92:6802/3740442374,v1:172.21.15.92:6803/3740442374] boot 2024-06-21T01:53:31.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[64276]: osdmap e110: 8 total, 8 up, 8 in 2024-06-21T01:53:31.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:53:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[66756]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:53:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[66756]: osd.0 [v2:172.21.15.92:6802/3740442374,v1:172.21.15.92:6803/3740442374] boot 2024-06-21T01:53:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[66756]: osdmap e110: 8 total, 8 up, 8 in 2024-06-21T01:53:31.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:30 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-21T01:53:31.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:31 smithi173 ceph-mon[57540]: osdmap e111: 8 total, 8 up, 8 in 2024-06-21T01:53:31.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:31 smithi173 ceph-mon[57540]: pgmap v48: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/723 objects degraded (10.650%) 2024-06-21T01:53:32.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:31 smithi092 ceph-mon[64276]: osdmap e111: 8 total, 8 up, 8 in 2024-06-21T01:53:32.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:31 smithi092 ceph-mon[64276]: pgmap v48: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/723 objects degraded (10.650%) 2024-06-21T01:53:32.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:31 smithi092 ceph-mon[66756]: osdmap e111: 8 total, 8 up, 8 in 2024-06-21T01:53:32.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:31 smithi092 ceph-mon[66756]: pgmap v48: 161 pgs: 36 active+undersized, 22 active+undersized+degraded, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 77/723 objects degraded (10.650%) 2024-06-21T01:53:33.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:53:33.016+0000 7f9965fe7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-06-21T01:53:33.887 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: pgmap v49: 161 pgs: 14 active+undersized, 7 active+undersized+degraded, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 32/723 objects degraded (4.426%) 2024-06-21T01:53:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:34.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:53:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded (PG_DEGRADED) 2024-06-21T01:53:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:53:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:34.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: pgmap v49: 161 pgs: 14 active+undersized, 7 active+undersized+degraded, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 32/723 objects degraded (4.426%) 2024-06-21T01:53:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:34.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:53:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded (PG_DEGRADED) 2024-06-21T01:53:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:53:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: pgmap v49: 161 pgs: 14 active+undersized, 7 active+undersized+degraded, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 32/723 objects degraded (4.426%) 2024-06-21T01:53:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:53:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded (PG_DEGRADED) 2024-06-21T01:53:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:53:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:34.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:35.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:34 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:35.542 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:34 smithi173 ceph-mon[57540]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-06-21T01:53:35.542 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:34 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:35.543 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:34 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:35.543 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:34 smithi092 ceph-mon[64276]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-06-21T01:53:35.543 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:34 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:35.543 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:34 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:35.544 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:34 smithi092 ceph-mon[66756]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-06-21T01:53:35.544 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:34 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:36.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:35 smithi173 ceph-mon[57540]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 769 B/s rd, 0 op/s 2024-06-21T01:53:36.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:35 smithi173 ceph-mon[57540]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded) 2024-06-21T01:53:36.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:35 smithi173 ceph-mon[57540]: Cluster is now healthy 2024-06-21T01:53:36.252 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:35 smithi092 ceph-mon[66756]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 769 B/s rd, 0 op/s 2024-06-21T01:53:36.253 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:35 smithi092 ceph-mon[66756]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded) 2024-06-21T01:53:36.253 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:35 smithi092 ceph-mon[66756]: Cluster is now healthy 2024-06-21T01:53:36.253 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:35 smithi092 ceph-mon[64276]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 769 B/s rd, 0 op/s 2024-06-21T01:53:36.253 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:35 smithi092 ceph-mon[64276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded) 2024-06-21T01:53:36.253 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:35 smithi092 ceph-mon[64276]: Cluster is now healthy 2024-06-21T01:53:38.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:37 smithi092 ceph-mon[64276]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-21T01:53:38.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:37 smithi092 ceph-mon[66756]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-21T01:53:38.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:37 smithi173 ceph-mon[57540]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-21T01:53:38.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:38 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-06-21T01:53:38.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:38 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:39.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:53:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:53:39] "GET /metrics HTTP/1.1" 200 38392 "" "Prometheus/2.33.4" 2024-06-21T01:53:40.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:39 smithi173 ceph-mon[57540]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:53:40.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:39 smithi092 ceph-mon[64276]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:53:40.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:39 smithi092 ceph-mon[66756]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:53:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:41 smithi173 ceph-mon[57540]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-06-21T01:53:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:42.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:41 smithi092 ceph-mon[64276]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-06-21T01:53:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:42.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:41 smithi092 ceph-mon[66756]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-06-21T01:53:42.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:43.929 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:43 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:44.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:43 smithi092 ceph-mon[64276]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:44.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:43 smithi092 ceph-mon[66756]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:44.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:43 smithi173 ceph-mon[57540]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:45.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:44 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:44 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:45.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:44 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:46.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:45 smithi173 ceph-mon[57540]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:45 smithi092 ceph-mon[64276]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:46.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:45 smithi092 ceph-mon[66756]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:47.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:46 smithi173 ceph-mon[57540]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:46 smithi092 ceph-mon[64276]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:47.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:46 smithi092 ceph-mon[66756]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:48.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:48 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:48.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:48 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:48.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:48 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:48.601 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:53:48.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:48 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:48.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-06-21T01:53:48.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:48 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:49.284 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:53:49.284 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (2m) 20s ago 7m 14.8M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:53:49.284 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (2m) 62s ago 6m 46.8M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (2m) 20s ago 6m 52.2M - 3.5 e1d6a67b021e 3192bc59456b 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (118s) 62s ago 11m 438M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (2m) 20s ago 13m 496M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (105s) 20s ago 13m 44.9M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (71s) 62s ago 12m 22.2M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (84s) 20s ago 12m 36.3M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (2m) 20s ago 8m 15.5M - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (2m) 62s ago 7m 20.3M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (24s) 20s ago 11m 11.8M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 7c252b0f2dfd 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (11m) 20s ago 11m 56.5M 2736M 17.2.0 e1d6a67b021e e0bf7c0600e5 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (10m) 20s ago 10m 49.9M 2736M 17.2.0 e1d6a67b021e 157fb16866c9 2024-06-21T01:53:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (10m) 20s ago 10m 52.1M 2736M 17.2.0 e1d6a67b021e e5567e44eaec 2024-06-21T01:53:49.286 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (9m) 62s ago 9m 56.0M 2992M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:53:49.286 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (9m) 62s ago 9m 54.0M 2992M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:53:49.286 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (9m) 62s ago 9m 48.9M 2992M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:53:49.286 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (8m) 62s ago 8m 53.2M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:53:49.286 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (2m) 62s ago 8m 60.0M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:53:49.286 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (6m) 20s ago 6m 76.4M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:53:49.286 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (6m) 62s ago 6m 72.6M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:53:49.286 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (6m) 20s ago 6m 74.0M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:53:49.286 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (6m) 62s ago 6m 73.8M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:53:49.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:53:49 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:53:49] "GET /metrics HTTP/1.1" 200 38398 "" "Prometheus/2.33.4" 2024-06-21T01:53:49.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:49 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:49.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:49 smithi092 ceph-mon[66756]: Upgrade: osd.1 is safe to restart 2024-06-21T01:53:49.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:49 smithi092 ceph-mon[66756]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:49.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:49 smithi092 ceph-mon[66756]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:49.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:49 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:49.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:49 smithi092 ceph-mon[64276]: Upgrade: osd.1 is safe to restart 2024-06-21T01:53:49.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:49 smithi092 ceph-mon[64276]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:49.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:49 smithi092 ceph-mon[64276]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:49.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:49 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-21T01:53:49.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:49 smithi173 ceph-mon[57540]: Upgrade: osd.1 is safe to restart 2024-06-21T01:53:49.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:49 smithi173 ceph-mon[57540]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:49.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:49 smithi173 ceph-mon[57540]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:49.690 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:53:49.690 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:53:49.690 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:53:49.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:53:49.690 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:53:49.690 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 1 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 6 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:53:49.691 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:53:50.036 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:53:50.037 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:53:50.037 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:53:50.037 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:53:50.037 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:53:50.037 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr", 2024-06-21T01:53:50.037 INFO:teuthology.orchestra.run.smithi092.stdout: "mon" 2024-06-21T01:53:50.037 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:53:50.037 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "6/23 daemons upgraded", 2024-06-21T01:53:50.037 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading osd daemons", 2024-06-21T01:53:50.037 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:53:50.037 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:53:50.433 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:53:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[64276]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[64276]: from='client.44191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[64276]: Upgrade: Updating osd.1 2024-06-21T01:53:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-21T01:53:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[64276]: Deploying daemon osd.1 on smithi092 2024-06-21T01:53:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/603846661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:50.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/51342501' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:53:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[66756]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[66756]: from='client.44191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[66756]: Upgrade: Updating osd.1 2024-06-21T01:53:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-21T01:53:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[66756]: Deploying daemon osd.1 on smithi092 2024-06-21T01:53:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/603846661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:50 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/51342501' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:53:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:50 smithi173 ceph-mon[57540]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:50 smithi173 ceph-mon[57540]: from='client.44191 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:50 smithi173 ceph-mon[57540]: Upgrade: Updating osd.1 2024-06-21T01:53:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:50 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:50.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:50 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-21T01:53:50.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:50 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:53:50.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:50 smithi173 ceph-mon[57540]: Deploying daemon osd.1 on smithi092 2024-06-21T01:53:50.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:50 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/603846661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:53:50.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:50 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/51342501' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:53:51.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:51 smithi092 ceph-mon[64276]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:51.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:51 smithi092 ceph-mon[64276]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:51.849 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:51 smithi092 ceph-mon[66756]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:51.849 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:51 smithi092 ceph-mon[66756]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:51.880 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:51 smithi173 ceph-mon[57540]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:53:51.880 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:51 smithi173 ceph-mon[57540]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:53:52.737 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:52 smithi092 systemd[1]: Stopping Ceph osd.1 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:53:53.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:52 smithi092 ceph-mon[64276]: osd.1 marked itself down and dead 2024-06-21T01:53:53.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:52 smithi092 ceph-mon[66756]: osd.1 marked itself down and dead 2024-06-21T01:53:53.101 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:52 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[35593]: 2024-06-21T01:53:52.772+0000 7f021ad03700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:53:53.101 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:52 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[35593]: 2024-06-21T01:53:52.772+0000 7f021ad03700 -1 osd.1 111 *** Got signal Terminated *** 2024-06-21T01:53:53.101 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:52 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[35593]: 2024-06-21T01:53:52.772+0000 7f021ad03700 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T01:53:53.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:52 smithi173 ceph-mon[57540]: osd.1 marked itself down and dead 2024-06-21T01:53:53.537 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:53 smithi092 podman[72086]: 2024-06-21 01:53:53.222203961 +0000 UTC m=+0.566207660 container died e0bf7c0600e5937edf12f3ac83e1d9da4f08263043b1445de40de576fe9fdb65 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vcs-type=git, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, maintainer=Guillaume Abrioux , ceph=True, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD) 2024-06-21T01:53:53.843 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:53 smithi092 podman[72086]: 2024-06-21 01:53:53.640549886 +0000 UTC m=+0.984553586 container cleanup e0bf7c0600e5937edf12f3ac83e1d9da4f08263043b1445de40de576fe9fdb65 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, name=centos-stream, io.openshift.tags=base centos centos-stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, architecture=x86_64) 2024-06-21T01:53:53.844 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:53 smithi092 bash[72086]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1 2024-06-21T01:53:53.845 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:54.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:53 smithi092 ceph-mon[64276]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:53:54.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:53 smithi092 ceph-mon[64276]: osdmap e112: 8 total, 7 up, 8 in 2024-06-21T01:53:54.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:53 smithi092 ceph-mon[64276]: pgmap v60: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:53:54.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:53 smithi092 ceph-mon[64276]: osdmap e113: 8 total, 7 up, 8 in 2024-06-21T01:53:54.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:53 smithi092 ceph-mon[66756]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:53:54.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:53 smithi092 ceph-mon[66756]: osdmap e112: 8 total, 7 up, 8 in 2024-06-21T01:53:54.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:53 smithi092 ceph-mon[66756]: pgmap v60: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:53:54.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:53 smithi092 ceph-mon[66756]: osdmap e113: 8 total, 7 up, 8 in 2024-06-21T01:53:54.103 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:53 smithi092 podman[72113]: 2024-06-21 01:53:53.876537207 +0000 UTC m=+0.653451479 container remove e0bf7c0600e5937edf12f3ac83e1d9da4f08263043b1445de40de576fe9fdb65 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_CLEAN=True, io.openshift.expose-services=, maintainer=Guillaume Abrioux , distribution-scope=public, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, version=8, io.buildah.version=1.19.8, release=754, vcs-type=git, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-06-21T01:53:54.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:53 smithi173 ceph-mon[57540]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:53:54.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:53 smithi173 ceph-mon[57540]: osdmap e112: 8 total, 7 up, 8 in 2024-06-21T01:53:54.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:53 smithi173 ceph-mon[57540]: pgmap v60: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:53:54.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:53 smithi173 ceph-mon[57540]: osdmap e113: 8 total, 7 up, 8 in 2024-06-21T01:53:54.583 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:54 smithi092 podman[72203]: 2024-06-21 01:53:54.209378389 +0000 UTC m=+0.017230585 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:53:54.583 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:54 smithi092 podman[72203]: 2024-06-21 01:53:54.351397394 +0000 UTC m=+0.159249582 container create 761c681b30df3bc01c1b5eeed61fdc603c1378244b8935b66bf34c8900a44dc4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-deactivate, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T01:53:54.844 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:54 smithi092 podman[72203]: 2024-06-21 01:53:54.587136439 +0000 UTC m=+0.394988634 container init 761c681b30df3bc01c1b5eeed61fdc603c1378244b8935b66bf34c8900a44dc4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-21T01:53:54.844 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:54 smithi092 podman[72203]: 2024-06-21 01:53:54.590745494 +0000 UTC m=+0.398597683 container start 761c681b30df3bc01c1b5eeed61fdc603c1378244b8935b66bf34c8900a44dc4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618) 2024-06-21T01:53:54.845 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:54 smithi092 podman[72203]: 2024-06-21 01:53:54.687381784 +0000 UTC m=+0.495233983 container attach 761c681b30df3bc01c1b5eeed61fdc603c1378244b8935b66bf34c8900a44dc4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240618) 2024-06-21T01:53:55.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:54 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:55.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:54 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:55.101 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:54 smithi092 podman[72203]: 2024-06-21 01:53:54.84928181 +0000 UTC m=+0.657133997 container died 761c681b30df3bc01c1b5eeed61fdc603c1378244b8935b66bf34c8900a44dc4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-21T01:53:55.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:54 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:53:55.601 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:55 smithi092 podman[72226]: 2024-06-21 01:53:55.407496845 +0000 UTC m=+0.552680474 container remove 761c681b30df3bc01c1b5eeed61fdc603c1378244b8935b66bf34c8900a44dc4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, GIT_CLEAN=True) 2024-06-21T01:53:55.601 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:55 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.1.service: Deactivated successfully. 2024-06-21T01:53:55.601 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:55 smithi092 systemd[1]: Stopped Ceph osd.1 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:53:55.601 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:55 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.1.service: Consumed 7.507s CPU time. 2024-06-21T01:53:55.991 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:55 smithi092 ceph-mon[64276]: pgmap v62: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:55.991 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:55 smithi092 ceph-mon[66756]: pgmap v62: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:55.992 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:55 smithi092 systemd[1]: Starting Ceph osd.1 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:53:55.992 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:55 smithi092 podman[72326]: 2024-06-21 01:53:55.84991421 +0000 UTC m=+0.108895712 container create 4177f47dbd9ec1418e5719005401142038971b2d9001260ee9fd1a60f5ca3c3e (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, ceph=True) 2024-06-21T01:53:55.992 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:55 smithi092 podman[72326]: 2024-06-21 01:53:55.758201418 +0000 UTC m=+0.017182931 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:53:56.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:55 smithi173 ceph-mon[57540]: pgmap v62: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:53:56.351 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 podman[72326]: 2024-06-21 01:53:56.119279717 +0000 UTC m=+0.378261240 container init 4177f47dbd9ec1418e5719005401142038971b2d9001260ee9fd1a60f5ca3c3e (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81) 2024-06-21T01:53:56.351 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 podman[72326]: 2024-06-21 01:53:56.122539663 +0000 UTC m=+0.381521173 container start 4177f47dbd9ec1418e5719005401142038971b2d9001260ee9fd1a60f5ca3c3e (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-06-21T01:53:56.351 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 podman[72326]: 2024-06-21 01:53:56.202591899 +0000 UTC m=+0.461573407 container attach 4177f47dbd9ec1418e5719005401142038971b2d9001260ee9fd1a60f5ca3c3e (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-21T01:53:56.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate[72341]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-21T01:53:56.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 bash[72326]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate[72341]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 bash[72326]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate[72341]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 bash[72326]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate[72341]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 bash[72326]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate[72341]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 bash[72326]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate[72341]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 bash[72326]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate[72341]: --> ceph-volume raw activate successful for osd ID: 1 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 bash[72326]: --> ceph-volume raw activate successful for osd ID: 1 2024-06-21T01:53:56.852 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:56 smithi092 podman[72326]: 2024-06-21 01:53:56.540708145 +0000 UTC m=+0.799689650 container died 4177f47dbd9ec1418e5719005401142038971b2d9001260ee9fd1a60f5ca3c3e (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-06-21T01:53:57.144 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:57.144 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:57.144 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:57.145 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:53:57.473 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:57 smithi092 podman[72474]: 2024-06-21 01:53:57.148300112 +0000 UTC m=+0.601826868 container remove 4177f47dbd9ec1418e5719005401142038971b2d9001260ee9fd1a60f5ca3c3e (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-release-c65be81, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:53:57.474 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:57 smithi092 podman[72534]: 2024-06-21 01:53:57.261601199 +0000 UTC m=+0.017558817 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:53:57.474 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:57 smithi092 podman[72534]: 2024-06-21 01:53:57.365883355 +0000 UTC m=+0.121841008 container create 9ad8cda613905109e0b5f37026dc3892f886148891938cb5f51ed41dfb55b855 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240618) 2024-06-21T01:53:57.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:57 smithi092 podman[72534]: 2024-06-21 01:53:57.601344907 +0000 UTC m=+0.357302530 container init 9ad8cda613905109e0b5f37026dc3892f886148891938cb5f51ed41dfb55b855 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, ceph=True, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-21T01:53:57.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:57 smithi092 podman[72534]: 2024-06-21 01:53:57.60458564 +0000 UTC m=+0.360543250 container start 9ad8cda613905109e0b5f37026dc3892f886148891938cb5f51ed41dfb55b855 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-release-c65be81, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0) 2024-06-21T01:53:57.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:57 smithi092 ceph-osd[72553]: -- 172.21.15.92:0/1768061796 <== mon.0 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558040d8fa40 con 0x558040d6dc00 2024-06-21T01:53:57.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:57 smithi092 bash[72534]: 9ad8cda613905109e0b5f37026dc3892f886148891938cb5f51ed41dfb55b855 2024-06-21T01:53:57.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:57 smithi092 systemd[1]: Started Ceph osd.1 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:53:58.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:57 smithi173 ceph-mon[57540]: pgmap v63: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-21T01:53:58.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:57 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:58.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:57 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:58.336 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:53:58 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[72549]: 2024-06-21T01:53:58.150+0000 7fbd6f73e640 -1 Falling back to public interface 2024-06-21T01:53:58.336 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:57 smithi092 ceph-mon[64276]: pgmap v63: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-21T01:53:58.336 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:57 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:58.336 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:57 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:58.336 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:57 smithi092 ceph-mon[66756]: pgmap v63: 161 pgs: 7 peering, 19 stale+active+clean, 135 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-21T01:53:58.336 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:57 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:58.337 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:57 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:53:58.943 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:53:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:53:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:53:59.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:58 smithi173 ceph-mon[57540]: pgmap v64: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-06-21T01:53:59.237 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:58 smithi092 ceph-mon[64276]: pgmap v64: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-06-21T01:53:59.237 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:58 smithi092 ceph-mon[66756]: pgmap v64: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-06-21T01:53:59.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:53:59 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:53:59] "GET /metrics HTTP/1.1" 200 38403 "" "Prometheus/2.33.4" 2024-06-21T01:54:00.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:53:59 smithi173 ceph-mon[57540]: Health check failed: Degraded data redundancy: 100/723 objects degraded (13.831%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:00.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:53:59 smithi092 ceph-mon[64276]: Health check failed: Degraded data redundancy: 100/723 objects degraded (13.831%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:00.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:53:59 smithi092 ceph-mon[66756]: Health check failed: Degraded data redundancy: 100/723 objects degraded (13.831%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:00.949 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:54:00 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[72549]: 2024-06-21T01:54:00.850+0000 7fbd6f73e640 -1 osd.1 0 read_superblock omap replica is missing. 2024-06-21T01:54:00.949 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:54:00 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[72549]: 2024-06-21T01:54:00.880+0000 7fbd6f73e640 -1 osd.1 111 log_to_monitors true 2024-06-21T01:54:01.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:00 smithi173 ceph-mon[57540]: from='osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-21T01:54:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:00 smithi173 ceph-mon[57540]: pgmap v65: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-06-21T01:54:01.351 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 01:54:00 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[72549]: 2024-06-21T01:54:00.977+0000 7fbd674e9640 -1 osd.1 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:54:01.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:00 smithi092 ceph-mon[64276]: from='osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-21T01:54:01.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:00 smithi092 ceph-mon[64276]: pgmap v65: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-06-21T01:54:01.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:00 smithi092 ceph-mon[66756]: from='osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-21T01:54:01.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:00 smithi092 ceph-mon[66756]: pgmap v65: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-06-21T01:54:02.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:01 smithi173 ceph-mon[57540]: from='osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-21T01:54:02.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:01 smithi173 ceph-mon[57540]: osdmap e114: 8 total, 7 up, 8 in 2024-06-21T01:54:02.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:01 smithi173 ceph-mon[57540]: from='osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:54:02.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:01 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:02.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:01 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:02.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:01 smithi092 ceph-mon[64276]: from='osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-21T01:54:02.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:01 smithi092 ceph-mon[64276]: osdmap e114: 8 total, 7 up, 8 in 2024-06-21T01:54:02.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:01 smithi092 ceph-mon[64276]: from='osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:54:02.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:01 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:02.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:01 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:02.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:01 smithi092 ceph-mon[66756]: from='osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-21T01:54:02.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:01 smithi092 ceph-mon[66756]: osdmap e114: 8 total, 7 up, 8 in 2024-06-21T01:54:02.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:01 smithi092 ceph-mon[66756]: from='osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:54:02.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:01 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:02.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:01 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:03.158 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[64276]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:54:03.159 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[64276]: osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674] boot 2024-06-21T01:54:03.159 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[64276]: osdmap e115: 8 total, 8 up, 8 in 2024-06-21T01:54:03.159 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:54:03.159 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:03.159 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:03.159 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[64276]: pgmap v68: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-06-21T01:54:03.159 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[64276]: osdmap e116: 8 total, 8 up, 8 in 2024-06-21T01:54:03.159 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[66756]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:54:03.159 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[66756]: osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674] boot 2024-06-21T01:54:03.159 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[66756]: osdmap e115: 8 total, 8 up, 8 in 2024-06-21T01:54:03.160 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:54:03.160 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:03.160 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:03.160 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[66756]: pgmap v68: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-06-21T01:54:03.160 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:02 smithi092 ceph-mon[66756]: osdmap e116: 8 total, 8 up, 8 in 2024-06-21T01:54:03.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:02 smithi173 ceph-mon[57540]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:54:03.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:02 smithi173 ceph-mon[57540]: osd.1 [v2:172.21.15.92:6810/3255796674,v1:172.21.15.92:6811/3255796674] boot 2024-06-21T01:54:03.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:02 smithi173 ceph-mon[57540]: osdmap e115: 8 total, 8 up, 8 in 2024-06-21T01:54:03.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:02 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-21T01:54:03.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:02 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:03.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:02 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:03.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:02 smithi173 ceph-mon[57540]: pgmap v68: 161 pgs: 41 active+undersized, 22 active+undersized+degraded, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 100/723 objects degraded (13.831%) 2024-06-21T01:54:03.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:02 smithi173 ceph-mon[57540]: osdmap e116: 8 total, 8 up, 8 in 2024-06-21T01:54:03.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:04.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:04 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:04.817 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:04 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:04.817 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:04 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:05.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:05 smithi092 ceph-mon[64276]: pgmap v70: 161 pgs: 36 active+undersized, 21 active+undersized+degraded, 104 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/723 objects degraded (13.001%) 2024-06-21T01:54:05.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:05 smithi092 ceph-mon[66756]: pgmap v70: 161 pgs: 36 active+undersized, 21 active+undersized+degraded, 104 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/723 objects degraded (13.001%) 2024-06-21T01:54:05.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:54:05 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:54:05.637+0000 7f9965fe7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-06-21T01:54:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:05 smithi173 ceph-mon[57540]: pgmap v70: 161 pgs: 36 active+undersized, 21 active+undersized+degraded, 104 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/723 objects degraded (13.001%) 2024-06-21T01:54:06.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 94/723 objects degraded (13.001%), 21 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:06.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:06.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:06.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:06.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:54:06.753 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 94/723 objects degraded (13.001%), 21 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:06.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:06.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:54:06.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:06.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:54:06.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:06.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:06.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:06.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:06.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:06.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-06-21T01:54:06.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:06.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 94/723 objects degraded (13.001%), 21 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-06-21T01:54:06.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:07.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:07 smithi092 ceph-mon[64276]: pgmap v71: 161 pgs: 23 active+undersized, 14 active+undersized+degraded, 124 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 57/723 objects degraded (7.884%) 2024-06-21T01:54:07.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:07 smithi092 ceph-mon[66756]: pgmap v71: 161 pgs: 23 active+undersized, 14 active+undersized+degraded, 124 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 57/723 objects degraded (7.884%) 2024-06-21T01:54:07.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:07 smithi173 ceph-mon[57540]: pgmap v71: 161 pgs: 23 active+undersized, 14 active+undersized+degraded, 124 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 57/723 objects degraded (7.884%) 2024-06-21T01:54:08.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:08.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-06-21T01:54:08.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:08.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:09.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:08 smithi092 ceph-mon[64276]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-21T01:54:09.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:08 smithi092 ceph-mon[64276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/723 objects degraded (7.884%), 14 pgs degraded) 2024-06-21T01:54:09.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:08 smithi092 ceph-mon[64276]: Cluster is now healthy 2024-06-21T01:54:09.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:08 smithi092 ceph-mon[66756]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-21T01:54:09.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:08 smithi092 ceph-mon[66756]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/723 objects degraded (7.884%), 14 pgs degraded) 2024-06-21T01:54:09.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:08 smithi092 ceph-mon[66756]: Cluster is now healthy 2024-06-21T01:54:09.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:08 smithi173 ceph-mon[57540]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-21T01:54:09.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:08 smithi173 ceph-mon[57540]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/723 objects degraded (7.884%), 14 pgs degraded) 2024-06-21T01:54:09.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:08 smithi173 ceph-mon[57540]: Cluster is now healthy 2024-06-21T01:54:09.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:54:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:54:09] "GET /metrics HTTP/1.1" 200 38403 "" "Prometheus/2.33.4" 2024-06-21T01:54:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:11 smithi173 ceph-mon[57540]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-06-21T01:54:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:12.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:11 smithi092 ceph-mon[66756]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-06-21T01:54:12.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:12.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:11 smithi092 ceph-mon[64276]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-06-21T01:54:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:13.929 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:13 smithi092 ceph-mon[64276]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:54:14.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:13 smithi092 ceph-mon[66756]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:54:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:13 smithi173 ceph-mon[57540]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:54:15.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:14 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:15.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:14 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:15.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:14 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:15 smithi173 ceph-mon[57540]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-06-21T01:54:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:15 smithi092 ceph-mon[64276]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-06-21T01:54:16.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:15 smithi092 ceph-mon[66756]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-06-21T01:54:17.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:16 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:17.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:16 smithi173 ceph-mon[57540]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:16 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:16 smithi092 ceph-mon[64276]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:17.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:16 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:17.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:16 smithi092 ceph-mon[66756]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:18.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:18.660Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-06-21T01:54:18.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:18.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:19.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:18 smithi092 ceph-mon[64276]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:19.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:18 smithi092 ceph-mon[66756]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:19.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:18 smithi173 ceph-mon[57540]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:19.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:54:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:54:19] "GET /metrics HTTP/1.1" 200 38402 "" "Prometheus/2.33.4" 2024-06-21T01:54:20.799 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:54:20.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:20 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:21.024 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:20 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:21.025 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:20 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:21.501 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:54:21.501 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (3m) 19s ago 7m 21.9M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (3m) 94s ago 7m 46.8M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (2m) 19s ago 6m 52.3M - 3.5 e1d6a67b021e 3192bc59456b 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (2m) 94s ago 12m 438M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (3m) 19s ago 13m 498M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (2m) 19s ago 14m 46.7M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (103s) 94s ago 12m 22.2M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (116s) 19s ago 12m 38.4M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (3m) 19s ago 8m 17.5M - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (3m) 94s ago 8m 20.3M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (56s) 19s ago 11m 51.0M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 7c252b0f2dfd 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (24s) 19s ago 11m 12.1M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 9ad8cda61390 2024-06-21T01:54:21.502 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (11m) 19s ago 11m 51.1M 2736M 17.2.0 e1d6a67b021e 157fb16866c9 2024-06-21T01:54:21.503 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (10m) 19s ago 10m 53.8M 2736M 17.2.0 e1d6a67b021e e5567e44eaec 2024-06-21T01:54:21.503 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (10m) 94s ago 10m 56.0M 2992M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:54:21.503 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (10m) 94s ago 10m 54.0M 2992M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:54:21.503 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (9m) 94s ago 9m 48.9M 2992M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:54:21.503 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (9m) 94s ago 9m 53.2M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:54:21.503 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (2m) 94s ago 8m 60.0M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:54:21.503 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (7m) 19s ago 7m 76.5M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:54:21.503 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (7m) 94s ago 6m 72.6M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:54:21.503 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (6m) 19s ago 6m 74.2M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:54:21.503 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (6m) 94s ago 6m 73.8M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:54:21.901 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:54:21.901 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:54:21.901 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:54:21.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-06-21T01:54:21.902 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 7 2024-06-21T01:54:21.903 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:54:21.903 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:54:21.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:21 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:21.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:21 smithi173 ceph-mon[57540]: Upgrade: osd.2 is safe to restart 2024-06-21T01:54:21.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:21 smithi173 ceph-mon[57540]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:21.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:21 smithi173 ceph-mon[57540]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:54:22.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:21 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:22.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:21 smithi092 ceph-mon[64276]: Upgrade: osd.2 is safe to restart 2024-06-21T01:54:22.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:21 smithi092 ceph-mon[64276]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:22.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:21 smithi092 ceph-mon[64276]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:54:22.041 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:21 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-21T01:54:22.041 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:21 smithi092 ceph-mon[66756]: Upgrade: osd.2 is safe to restart 2024-06-21T01:54:22.041 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:21 smithi092 ceph-mon[66756]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:22.041 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:21 smithi092 ceph-mon[66756]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:54:22.260 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:54:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:54:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:54:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:54:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:54:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr", 2024-06-21T01:54:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "mon" 2024-06-21T01:54:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:54:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "7/23 daemons upgraded", 2024-06-21T01:54:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading osd daemons", 2024-06-21T01:54:22.261 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:54:22.261 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:54:22.674 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:54:22.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[64276]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[64276]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/785617602' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-21T01:54:22.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:22.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/2367162874' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:54:22.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[66756]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:22.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[66756]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:22.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/785617602' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:22.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:22.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-21T01:54:22.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:22.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:22 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/2367162874' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:54:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:22 smithi173 ceph-mon[57540]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:22 smithi173 ceph-mon[57540]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:22 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/785617602' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:22 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:22 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-21T01:54:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:22 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:22.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:22 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/2367162874' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:54:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:23 smithi092 ceph-mon[64276]: Upgrade: Updating osd.2 2024-06-21T01:54:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:23 smithi092 ceph-mon[64276]: Deploying daemon osd.2 on smithi092 2024-06-21T01:54:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:23 smithi092 ceph-mon[64276]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:23.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:23 smithi092 ceph-mon[64276]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:23.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:23 smithi092 ceph-mon[66756]: Upgrade: Updating osd.2 2024-06-21T01:54:23.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:23 smithi092 ceph-mon[66756]: Deploying daemon osd.2 on smithi092 2024-06-21T01:54:23.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:23 smithi092 ceph-mon[66756]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:23.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:23 smithi092 ceph-mon[66756]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:23.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:23 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:23 smithi173 ceph-mon[57540]: Upgrade: Updating osd.2 2024-06-21T01:54:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:23 smithi173 ceph-mon[57540]: Deploying daemon osd.2 on smithi092 2024-06-21T01:54:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:23 smithi173 ceph-mon[57540]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:23.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:23 smithi173 ceph-mon[57540]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:24.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:24 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:25.009 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:24 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:25.009 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:24 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:25.601 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:25 smithi092 systemd[1]: Stopping Ceph osd.2 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:54:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:25 smithi173 ceph-mon[57540]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:54:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:25 smithi173 ceph-mon[57540]: osd.2 marked itself down and dead 2024-06-21T01:54:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:25 smithi092 ceph-mon[64276]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:54:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:25 smithi092 ceph-mon[64276]: osd.2 marked itself down and dead 2024-06-21T01:54:26.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:25 smithi092 ceph-mon[66756]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:54:26.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:25 smithi092 ceph-mon[66756]: osd.2 marked itself down and dead 2024-06-21T01:54:26.101 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:25 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[38495]: 2024-06-21T01:54:25.632+0000 7f58fb4bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:54:26.102 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:25 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[38495]: 2024-06-21T01:54:25.632+0000 7f58fb4bc700 -1 osd.2 116 *** Got signal Terminated *** 2024-06-21T01:54:26.102 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:25 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[38495]: 2024-06-21T01:54:25.632+0000 7f58fb4bc700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T01:54:26.481 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:26 smithi092 podman[74534]: 2024-06-21 01:54:26.164520519 +0000 UTC m=+0.656857175 container died 157fb16866c900757202089274d5970220e6c96c57b8278b1fcb9ea5e6776cb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_BRANCH=HEAD, version=8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., distribution-scope=public, release=754, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-06-21T01:54:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:26 smithi092 ceph-mon[64276]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:54:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:26 smithi092 ceph-mon[64276]: osdmap e117: 8 total, 7 up, 8 in 2024-06-21T01:54:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:26.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:26 smithi092 ceph-mon[66756]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:54:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:26 smithi092 ceph-mon[66756]: osdmap e117: 8 total, 7 up, 8 in 2024-06-21T01:54:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:26.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:26.852 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:26 smithi092 podman[74534]: 2024-06-21 01:54:26.633353029 +0000 UTC m=+1.125689726 container cleanup 157fb16866c900757202089274d5970220e6c96c57b8278b1fcb9ea5e6776cb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2, io.buildah.version=1.19.8, vcs-type=git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, GIT_BRANCH=HEAD, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-06-21T01:54:26.852 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:26 smithi092 bash[74534]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2 2024-06-21T01:54:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:26 smithi173 ceph-mon[57540]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:54:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:26 smithi173 ceph-mon[57540]: osdmap e117: 8 total, 7 up, 8 in 2024-06-21T01:54:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:27.223 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:26 smithi092 podman[74550]: 2024-06-21 01:54:26.910101277 +0000 UTC m=+0.740013246 container remove 157fb16866c900757202089274d5970220e6c96c57b8278b1fcb9ea5e6776cb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , name=centos-stream, version=8, ceph=True, io.buildah.version=1.19.8, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, release=754, distribution-scope=public, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870) 2024-06-21T01:54:27.501 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:27 smithi092 podman[74639]: 2024-06-21 01:54:27.227500142 +0000 UTC m=+0.110495650 container create 08ce732a5cd6e2771bf8debadd655339f6f9194cbb7da0a1360b9d61580ec082 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T01:54:27.501 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:27 smithi092 podman[74639]: 2024-06-21 01:54:27.134658622 +0000 UTC m=+0.017654130 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:54:27.501 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:27 smithi092 podman[74639]: 2024-06-21 01:54:27.446724241 +0000 UTC m=+0.329719744 container init 08ce732a5cd6e2771bf8debadd655339f6f9194cbb7da0a1360b9d61580ec082 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-21T01:54:27.501 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:27 smithi092 podman[74639]: 2024-06-21 01:54:27.450094423 +0000 UTC m=+0.333089931 container start 08ce732a5cd6e2771bf8debadd655339f6f9194cbb7da0a1360b9d61580ec082 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-06-21T01:54:27.501 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:27 smithi092 podman[74639]: 2024-06-21 01:54:27.504836868 +0000 UTC m=+0.387832372 container attach 08ce732a5cd6e2771bf8debadd655339f6f9194cbb7da0a1360b9d61580ec082 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618) 2024-06-21T01:54:27.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:27 smithi092 ceph-mon[64276]: osdmap e118: 8 total, 7 up, 8 in 2024-06-21T01:54:27.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:27 smithi092 ceph-mon[64276]: pgmap v83: 161 pgs: 21 active+undersized, 6 peering, 2 stale+active+clean, 12 active+undersized+degraded, 120 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 39/723 objects degraded (5.394%) 2024-06-21T01:54:27.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:27 smithi092 ceph-mon[66756]: osdmap e118: 8 total, 7 up, 8 in 2024-06-21T01:54:27.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:27 smithi092 ceph-mon[66756]: pgmap v83: 161 pgs: 21 active+undersized, 6 peering, 2 stale+active+clean, 12 active+undersized+degraded, 120 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 39/723 objects degraded (5.394%) 2024-06-21T01:54:27.852 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:27 smithi092 podman[74639]: 2024-06-21 01:54:27.713793791 +0000 UTC m=+0.596789299 container died 08ce732a5cd6e2771bf8debadd655339f6f9194cbb7da0a1360b9d61580ec082 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240618, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS) 2024-06-21T01:54:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:27 smithi173 ceph-mon[57540]: osdmap e118: 8 total, 7 up, 8 in 2024-06-21T01:54:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:27 smithi173 ceph-mon[57540]: pgmap v83: 161 pgs: 21 active+undersized, 6 peering, 2 stale+active+clean, 12 active+undersized+degraded, 120 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 39/723 objects degraded (5.394%) 2024-06-21T01:54:28.602 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:28 smithi092 podman[74662]: 2024-06-21 01:54:28.350576792 +0000 UTC m=+0.630864402 container remove 08ce732a5cd6e2771bf8debadd655339f6f9194cbb7da0a1360b9d61580ec082 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81) 2024-06-21T01:54:28.602 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:28 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.2.service: Deactivated successfully. 2024-06-21T01:54:28.602 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:28 smithi092 systemd[1]: Stopped Ceph osd.2 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:54:28.602 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:28 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.2.service: Consumed 6.192s CPU time. 2024-06-21T01:54:28.602 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:28 smithi092 systemd[1]: Starting Ceph osd.2 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:54:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:28 smithi173 ceph-mon[57540]: Health check failed: Degraded data redundancy: 39/723 objects degraded (5.394%), 12 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:28.979 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:28 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:29.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:28 smithi092 ceph-mon[64276]: Health check failed: Degraded data redundancy: 39/723 objects degraded (5.394%), 12 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:29.025 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:28 smithi092 ceph-mon[66756]: Health check failed: Degraded data redundancy: 39/723 objects degraded (5.394%), 12 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:29.025 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:28 smithi092 podman[74761]: 2024-06-21 01:54:28.690278965 +0000 UTC m=+0.017440442 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:54:29.026 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:28 smithi092 podman[74761]: 2024-06-21 01:54:28.809943532 +0000 UTC m=+0.137105047 container create 1b19c2ac235e8d2f9d23feab1bb647b2ff77408fe5e99c4a7eccdcf510621f01 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T01:54:29.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:54:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:54:29] "GET /metrics HTTP/1.1" 200 38414 "" "Prometheus/2.33.4" 2024-06-21T01:54:29.351 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 podman[74761]: 2024-06-21 01:54:29.02935584 +0000 UTC m=+0.356517329 container init 1b19c2ac235e8d2f9d23feab1bb647b2ff77408fe5e99c4a7eccdcf510621f01 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T01:54:29.351 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 podman[74761]: 2024-06-21 01:54:29.03324656 +0000 UTC m=+0.360408040 container start 1b19c2ac235e8d2f9d23feab1bb647b2ff77408fe5e99c4a7eccdcf510621f01 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T01:54:29.351 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 podman[74761]: 2024-06-21 01:54:29.112212587 +0000 UTC m=+0.439374060 container attach 1b19c2ac235e8d2f9d23feab1bb647b2ff77408fe5e99c4a7eccdcf510621f01 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-21T01:54:29.738 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate[74777]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 bash[74761]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate[74777]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 bash[74761]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate[74777]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 bash[74761]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate[74777]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 bash[74761]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate[74777]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 bash[74761]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate[74777]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 bash[74761]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-21T01:54:29.739 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate[74777]: --> ceph-volume raw activate successful for osd ID: 2 2024-06-21T01:54:29.740 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 bash[74761]: --> ceph-volume raw activate successful for osd ID: 2 2024-06-21T01:54:29.740 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:29 smithi092 podman[74761]: 2024-06-21 01:54:29.450074811 +0000 UTC m=+0.777236302 container died 1b19c2ac235e8d2f9d23feab1bb647b2ff77408fe5e99c4a7eccdcf510621f01 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-21T01:54:30.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:29 smithi092 ceph-mon[64276]: pgmap v84: 161 pgs: 27 active+undersized, 6 peering, 13 active+undersized+degraded, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/723 objects degraded (6.224%) 2024-06-21T01:54:30.029 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:29 smithi092 ceph-mon[66756]: pgmap v84: 161 pgs: 27 active+undersized, 6 peering, 13 active+undersized+degraded, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/723 objects degraded (6.224%) 2024-06-21T01:54:30.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:29 smithi173 ceph-mon[57540]: pgmap v84: 161 pgs: 27 active+undersized, 6 peering, 13 active+undersized+degraded, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/723 objects degraded (6.224%) 2024-06-21T01:54:30.351 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:30 smithi092 podman[74895]: 2024-06-21 01:54:30.032888876 +0000 UTC m=+0.577443566 container remove 1b19c2ac235e8d2f9d23feab1bb647b2ff77408fe5e99c4a7eccdcf510621f01 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-activate, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-release-c65be81, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-21T01:54:30.351 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:30 smithi092 podman[74953]: 2024-06-21 01:54:30.146074683 +0000 UTC m=+0.017624935 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:54:30.351 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:30 smithi092 podman[74953]: 2024-06-21 01:54:30.250347181 +0000 UTC m=+0.121897467 container create 0fe024e7dad9d0e0f5e1e50b66d27e0d673aa64b150ddccc695cea4c780b9662 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, ceph=True) 2024-06-21T01:54:30.854 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:30 smithi092 podman[74953]: 2024-06-21 01:54:30.469296258 +0000 UTC m=+0.340846514 container init 0fe024e7dad9d0e0f5e1e50b66d27e0d673aa64b150ddccc695cea4c780b9662 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240618, ceph=True) 2024-06-21T01:54:30.854 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:30 smithi092 podman[74953]: 2024-06-21 01:54:30.472428629 +0000 UTC m=+0.343978872 container start 0fe024e7dad9d0e0f5e1e50b66d27e0d673aa64b150ddccc695cea4c780b9662 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81) 2024-06-21T01:54:30.854 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:30 smithi092 ceph-osd[74972]: -- 172.21.15.92:0/128488638 <== mon.0 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5598bea81a40 con 0x5598bf840400 2024-06-21T01:54:30.854 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:30 smithi092 bash[74953]: 0fe024e7dad9d0e0f5e1e50b66d27e0d673aa64b150ddccc695cea4c780b9662 2024-06-21T01:54:30.854 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:30 smithi092 systemd[1]: Started Ceph osd.2 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:54:31.351 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:31 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[74968]: 2024-06-21T01:54:31.011+0000 7f6c9a86e640 -1 Falling back to public interface 2024-06-21T01:54:31.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:31 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:31.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:31 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:31.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:31 smithi173 ceph-mon[57540]: pgmap v85: 161 pgs: 27 active+undersized, 6 peering, 13 active+undersized+degraded, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 45/723 objects degraded (6.224%) 2024-06-21T01:54:32.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:31 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:32.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:31 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:32.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:31 smithi092 ceph-mon[64276]: pgmap v85: 161 pgs: 27 active+undersized, 6 peering, 13 active+undersized+degraded, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 45/723 objects degraded (6.224%) 2024-06-21T01:54:32.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:31 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:32.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:31 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:32.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:31 smithi092 ceph-mon[66756]: pgmap v85: 161 pgs: 27 active+undersized, 6 peering, 13 active+undersized+degraded, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 45/723 objects degraded (6.224%) 2024-06-21T01:54:33.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:32 smithi092 ceph-mon[64276]: pgmap v86: 161 pgs: 31 active+undersized, 15 active+undersized+degraded, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/723 objects degraded (9.267%) 2024-06-21T01:54:33.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:32 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 67/723 objects degraded (9.267%), 15 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:33.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:32 smithi092 ceph-mon[66756]: pgmap v86: 161 pgs: 31 active+undersized, 15 active+undersized+degraded, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/723 objects degraded (9.267%) 2024-06-21T01:54:33.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:32 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 67/723 objects degraded (9.267%), 15 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:33.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:32 smithi173 ceph-mon[57540]: pgmap v86: 161 pgs: 31 active+undersized, 15 active+undersized+degraded, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/723 objects degraded (9.267%) 2024-06-21T01:54:33.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:32 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 67/723 objects degraded (9.267%), 15 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:33.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:33 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:33.989 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[74968]: 2024-06-21T01:54:33.721+0000 7f6c9a86e640 -1 osd.2 0 read_superblock omap replica is missing. 2024-06-21T01:54:33.989 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:33 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[74968]: 2024-06-21T01:54:33.741+0000 7f6c9a86e640 -1 osd.2 116 log_to_monitors true 2024-06-21T01:54:34.304 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:33 smithi092 ceph-mon[64276]: from='osd.2 [v2:172.21.15.92:6818/816826204,v1:172.21.15.92:6819/816826204]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-21T01:54:34.304 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:33 smithi092 ceph-mon[64276]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-21T01:54:34.304 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:33 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:34.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:33 smithi092 ceph-mon[66756]: from='osd.2 [v2:172.21.15.92:6818/816826204,v1:172.21.15.92:6819/816826204]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-21T01:54:34.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:33 smithi092 ceph-mon[66756]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-21T01:54:34.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:33 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:34.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:33 smithi173 ceph-mon[57540]: from='osd.2 [v2:172.21.15.92:6818/816826204,v1:172.21.15.92:6819/816826204]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-21T01:54:34.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:33 smithi173 ceph-mon[57540]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-21T01:54:34.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:33 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:34.601 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[74968]: 2024-06-21T01:54:34.464+0000 7f6c8e611640 -1 osd.2 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:54:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[66756]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-21T01:54:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[66756]: from='osd.2 [v2:172.21.15.92:6818/816826204,v1:172.21.15.92:6819/816826204]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:54:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[66756]: osdmap e119: 8 total, 7 up, 8 in 2024-06-21T01:54:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[66756]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:54:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[66756]: from='osd.2 ' entity='osd.2' 2024-06-21T01:54:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:35.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:35.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[66756]: pgmap v88: 161 pgs: 31 active+undersized, 15 active+undersized+degraded, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/723 objects degraded (9.267%) 2024-06-21T01:54:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[64276]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-21T01:54:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[64276]: from='osd.2 [v2:172.21.15.92:6818/816826204,v1:172.21.15.92:6819/816826204]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:54:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[64276]: osdmap e119: 8 total, 7 up, 8 in 2024-06-21T01:54:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[64276]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:54:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[64276]: from='osd.2 ' entity='osd.2' 2024-06-21T01:54:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:34 smithi092 ceph-mon[64276]: pgmap v88: 161 pgs: 31 active+undersized, 15 active+undersized+degraded, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/723 objects degraded (9.267%) 2024-06-21T01:54:35.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:34 smithi173 ceph-mon[57540]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-21T01:54:35.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:34 smithi173 ceph-mon[57540]: from='osd.2 [v2:172.21.15.92:6818/816826204,v1:172.21.15.92:6819/816826204]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:54:35.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:34 smithi173 ceph-mon[57540]: osdmap e119: 8 total, 7 up, 8 in 2024-06-21T01:54:35.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:34 smithi173 ceph-mon[57540]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:54:35.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:34 smithi173 ceph-mon[57540]: from='osd.2 ' entity='osd.2' 2024-06-21T01:54:35.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:34 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:35.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:34 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:35.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:34 smithi173 ceph-mon[57540]: pgmap v88: 161 pgs: 31 active+undersized, 15 active+undersized+degraded, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/723 objects degraded (9.267%) 2024-06-21T01:54:36.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:36 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:36.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:36 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:36.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:36 smithi173 ceph-mon[57540]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:54:36.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:36 smithi173 ceph-mon[57540]: osd.2 [v2:172.21.15.92:6818/816826204,v1:172.21.15.92:6819/816826204] boot 2024-06-21T01:54:36.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:36 smithi173 ceph-mon[57540]: osdmap e120: 8 total, 8 up, 8 in 2024-06-21T01:54:36.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:36 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:54:36.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:36.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:36.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[64276]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:54:36.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[64276]: osd.2 [v2:172.21.15.92:6818/816826204,v1:172.21.15.92:6819/816826204] boot 2024-06-21T01:54:36.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[64276]: osdmap e120: 8 total, 8 up, 8 in 2024-06-21T01:54:36.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:54:36.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:36.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:36.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[66756]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:54:36.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[66756]: osd.2 [v2:172.21.15.92:6818/816826204,v1:172.21.15.92:6819/816826204] boot 2024-06-21T01:54:36.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[66756]: osdmap e120: 8 total, 8 up, 8 in 2024-06-21T01:54:36.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:36 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-21T01:54:37.731 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:37 smithi173 ceph-mon[57540]: osdmap e121: 8 total, 8 up, 8 in 2024-06-21T01:54:38.361 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:37 smithi173 ceph-mon[57540]: pgmap v91: 161 pgs: 4 peering, 15 active+undersized, 2 active+undersized+degraded, 140 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 7/723 objects degraded (0.968%) 2024-06-21T01:54:38.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:37 smithi092 ceph-mon[64276]: osdmap e121: 8 total, 8 up, 8 in 2024-06-21T01:54:38.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:37 smithi092 ceph-mon[64276]: pgmap v91: 161 pgs: 4 peering, 15 active+undersized, 2 active+undersized+degraded, 140 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 7/723 objects degraded (0.968%) 2024-06-21T01:54:38.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:37 smithi092 ceph-mon[66756]: osdmap e121: 8 total, 8 up, 8 in 2024-06-21T01:54:38.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:37 smithi092 ceph-mon[66756]: pgmap v91: 161 pgs: 4 peering, 15 active+undersized, 2 active+undersized+degraded, 140 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 7/723 objects degraded (0.968%) 2024-06-21T01:54:38.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:54:38 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:54:38.636+0000 7f9965fe7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-06-21T01:54:38.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:38 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-06-21T01:54:38.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:38 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:39.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:54:39] "GET /metrics HTTP/1.1" 200 38414 "" "Prometheus/2.33.4" 2024-06-21T01:54:39.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:39.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:39.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:39.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:54:39.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:39.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:54:39.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:39.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:39.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:39.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:39.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:39.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-06-21T01:54:39.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: pgmap v92: 161 pgs: 4 peering, 11 active+undersized, 1 active+undersized+degraded, 145 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/723 objects degraded (0.138%) 2024-06-21T01:54:39.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 7/723 objects degraded (0.968%), 2 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:39.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:39.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: pgmap v92: 161 pgs: 4 peering, 11 active+undersized, 1 active+undersized+degraded, 145 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/723 objects degraded (0.138%) 2024-06-21T01:54:39.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:39 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 7/723 objects degraded (0.968%), 2 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:39.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:39.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:39.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:39.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:54:39.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:39.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:54:39.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:39.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:39.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:39.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:39.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:39.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-06-21T01:54:39.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: pgmap v92: 161 pgs: 4 peering, 11 active+undersized, 1 active+undersized+degraded, 145 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/723 objects degraded (0.138%) 2024-06-21T01:54:39.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:39 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 7/723 objects degraded (0.968%), 2 pgs degraded (PG_DEGRADED) 2024-06-21T01:54:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:41 smithi173 ceph-mon[57540]: pgmap v93: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-06-21T01:54:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:42.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:41 smithi173 ceph-mon[57540]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/723 objects degraded (0.138%), 1 pg degraded) 2024-06-21T01:54:42.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:41 smithi173 ceph-mon[57540]: Cluster is now healthy 2024-06-21T01:54:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:41 smithi092 ceph-mon[64276]: pgmap v93: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-06-21T01:54:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:41 smithi092 ceph-mon[64276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/723 objects degraded (0.138%), 1 pg degraded) 2024-06-21T01:54:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:41 smithi092 ceph-mon[64276]: Cluster is now healthy 2024-06-21T01:54:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:41 smithi092 ceph-mon[66756]: pgmap v93: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-06-21T01:54:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:41 smithi092 ceph-mon[66756]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/723 objects degraded (0.138%), 1 pg degraded) 2024-06-21T01:54:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:41 smithi092 ceph-mon[66756]: Cluster is now healthy 2024-06-21T01:54:43.930 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:43 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:44.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:43 smithi092 ceph-mon[64276]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:44.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:43 smithi092 ceph-mon[66756]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:44.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:43 smithi173 ceph-mon[57540]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:45.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:44 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:44 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:45.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:44 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:46.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:45 smithi173 ceph-mon[57540]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:54:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:45 smithi092 ceph-mon[64276]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:54:46.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:45 smithi092 ceph-mon[66756]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:54:47.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:46 smithi173 ceph-mon[57540]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:54:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:46 smithi092 ceph-mon[64276]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:54:47.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:46 smithi092 ceph-mon[66756]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:54:48.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:48 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:48.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-06-21T01:54:48.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:48 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:48.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:49.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:48 smithi092 ceph-mon[64276]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:54:49.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:48 smithi092 ceph-mon[66756]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:54:49.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:48 smithi173 ceph-mon[57540]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:54:49.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:54:49 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:54:49] "GET /metrics HTTP/1.1" 200 38430 "" "Prometheus/2.33.4" 2024-06-21T01:54:51.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:50 smithi173 ceph-mon[57540]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:54:51.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:50 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:51.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:50 smithi092 ceph-mon[64276]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:54:51.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:50 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:51.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:50 smithi092 ceph-mon[66756]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:54:51.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:50 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:53.044 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:54:53.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:52 smithi092 ceph-mon[64276]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:53.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:52 smithi092 ceph-mon[66756]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:53.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:52 smithi173 ceph-mon[57540]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:53.695 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:54:53.695 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (3m) 19s ago 8m 21.9M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:54:53.695 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (3m) 2m ago 7m 46.8M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:54:53.695 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (3m) 19s ago 7m 52.4M - 3.5 e1d6a67b021e 3192bc59456b 2024-06-21T01:54:53.695 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (3m) 2m ago 12m 438M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:54:53.695 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (3m) 19s ago 14m 500M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:54:53.695 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (2m) 19s ago 14m 47.9M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (2m) 2m ago 13m 22.2M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (2m) 19s ago 13m 40.2M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (3m) 19s ago 9m 17.0M - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (3m) 2m ago 9m 20.3M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (89s) 19s ago 12m 51.4M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 7c252b0f2dfd 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (56s) 19s ago 12m 44.5M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 9ad8cda61390 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (23s) 19s ago 11m 11.7M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 0fe024e7dad9 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (11m) 19s ago 11m 54.5M 2736M 17.2.0 e1d6a67b021e e5567e44eaec 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (11m) 2m ago 10m 56.0M 2992M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (10m) 2m ago 10m 54.0M 2992M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (10m) 2m ago 10m 48.9M 2992M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (9m) 2m ago 9m 53.2M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (3m) 2m ago 9m 60.0M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:54:53.696 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (7m) 19s ago 7m 76.6M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:54:53.697 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (7m) 2m ago 7m 72.6M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:54:53.697 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (7m) 19s ago 7m 74.3M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:54:53.697 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (7m) 2m ago 7m 73.8M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:54:53.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:54.111 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:54:54.111 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:54:54.111 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:54:54.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:54:54.111 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:54:54.111 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:54:54.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:54:54.111 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:54:54.111 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-06-21T01:54:54.111 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:54:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:54:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:54:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:54:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:54:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:54:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-06-21T01:54:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 8 2024-06-21T01:54:54.112 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:54:54.112 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:54:54.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[64276]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:54.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[64276]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:54.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:54.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:54.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[64276]: Upgrade: osd.3 is safe to restart 2024-06-21T01:54:54.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[64276]: from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:54.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:54.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[66756]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:54.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[66756]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:54.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:54.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:54.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[66756]: Upgrade: osd.3 is safe to restart 2024-06-21T01:54:54.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[66756]: from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:54.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:53 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:54.454 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:54:54.454 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:54:54.454 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:54:54.454 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:54:54.454 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:54:54.454 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr", 2024-06-21T01:54:54.454 INFO:teuthology.orchestra.run.smithi092.stdout: "mon" 2024-06-21T01:54:54.455 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:54:54.455 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "8/23 daemons upgraded", 2024-06-21T01:54:54.455 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading osd daemons", 2024-06-21T01:54:54.455 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:54:54.455 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:54:54.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:53 smithi173 ceph-mon[57540]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:54.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:53 smithi173 ceph-mon[57540]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:54.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:54.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:53 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-21T01:54:54.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:53 smithi173 ceph-mon[57540]: Upgrade: osd.3 is safe to restart 2024-06-21T01:54:54.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:53 smithi173 ceph-mon[57540]: from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:54.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:53 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:54:54.853 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:54:55.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:55 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/2910261905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:55.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:55 smithi092 ceph-mon[66756]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:55.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:55 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/1124238356' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:54:55.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:55 smithi092 ceph-mon[66756]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:54:55.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:55 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/2910261905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:55.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:55 smithi092 ceph-mon[64276]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:55.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:55 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/1124238356' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:54:55.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:55 smithi092 ceph-mon[64276]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:54:55.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:55 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/2910261905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:54:55.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:55 smithi173 ceph-mon[57540]: from='client.44287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:54:55.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:55 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/1124238356' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:54:55.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:55 smithi173 ceph-mon[57540]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:54:56.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[64276]: Upgrade: Updating osd.3 2024-06-21T01:54:56.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:56.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-21T01:54:56.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:56.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[64276]: Deploying daemon osd.3 on smithi092 2024-06-21T01:54:56.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:56.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:56.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[66756]: Upgrade: Updating osd.3 2024-06-21T01:54:56.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:56.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-21T01:54:56.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:56.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[66756]: Deploying daemon osd.3 on smithi092 2024-06-21T01:54:56.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:56.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:56.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:56 smithi173 ceph-mon[57540]: Upgrade: Updating osd.3 2024-06-21T01:54:56.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:56.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-21T01:54:56.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:54:56.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:56 smithi173 ceph-mon[57540]: Deploying daemon osd.3 on smithi092 2024-06-21T01:54:56.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:54:56.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:54:57.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:57 smithi092 ceph-mon[64276]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:57.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:57 smithi092 ceph-mon[66756]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:57.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:57 smithi173 ceph-mon[57540]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:54:58.281 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:58 smithi092 systemd[1]: Stopping Ceph osd.3 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:54:58.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:58 smithi092 ceph-mon[64276]: osd.3 marked itself down and dead 2024-06-21T01:54:58.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:58 smithi092 ceph-mon[66756]: osd.3 marked itself down and dead 2024-06-21T01:54:58.601 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:58 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[41451]: 2024-06-21T01:54:58.282+0000 7f0edfd16700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:54:58.601 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:58 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[41451]: 2024-06-21T01:54:58.282+0000 7f0edfd16700 -1 osd.3 121 *** Got signal Terminated *** 2024-06-21T01:54:58.601 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:58 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[41451]: 2024-06-21T01:54:58.282+0000 7f0edfd16700 -1 osd.3 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T01:54:58.656 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:58 smithi173 ceph-mon[57540]: osd.3 marked itself down and dead 2024-06-21T01:54:58.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:54:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:54:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:54:58.989 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:58 smithi092 podman[76961]: 2024-06-21 01:54:58.715264358 +0000 UTC m=+0.528828458 container died e5567e44eaecacfc9db3431b4612bf3d3514ad21b6d8c3c0a2161bd6a98e9ab2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, release=754, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vendor=Red Hat, Inc., GIT_CLEAN=True, version=8) 2024-06-21T01:54:59.266 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:59 smithi092 podman[76961]: 2024-06-21 01:54:59.092140563 +0000 UTC m=+0.905704668 container cleanup e5567e44eaecacfc9db3431b4612bf3d3514ad21b6d8c3c0a2161bd6a98e9ab2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3, distribution-scope=public, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_BRANCH=HEAD, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream) 2024-06-21T01:54:59.266 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:59 smithi092 bash[76961]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3 2024-06-21T01:54:59.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:59 smithi092 ceph-mon[64276]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:54:59.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:59 smithi092 ceph-mon[64276]: osdmap e122: 8 total, 7 up, 8 in 2024-06-21T01:54:59.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:59 smithi092 ceph-mon[64276]: pgmap v103: 161 pgs: 14 peering, 21 stale+active+clean, 126 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-06-21T01:54:59.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:54:59 smithi092 ceph-mon[64276]: osdmap e123: 8 total, 7 up, 8 in 2024-06-21T01:54:59.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:59 smithi092 ceph-mon[66756]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:54:59.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:59 smithi092 ceph-mon[66756]: osdmap e122: 8 total, 7 up, 8 in 2024-06-21T01:54:59.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:59 smithi092 ceph-mon[66756]: pgmap v103: 161 pgs: 14 peering, 21 stale+active+clean, 126 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-06-21T01:54:59.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:54:59 smithi092 ceph-mon[66756]: osdmap e123: 8 total, 7 up, 8 in 2024-06-21T01:54:59.602 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:59 smithi092 podman[76977]: 2024-06-21 01:54:59.36007648 +0000 UTC m=+0.638964873 container remove e5567e44eaecacfc9db3431b4612bf3d3514ad21b6d8c3c0a2161bd6a98e9ab2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , name=centos-stream, ceph=True, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, RELEASE=HEAD, GIT_BRANCH=HEAD) 2024-06-21T01:54:59.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:54:59 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:54:59] "GET /metrics HTTP/1.1" 200 38428 "" "Prometheus/2.33.4" 2024-06-21T01:54:59.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:59 smithi173 ceph-mon[57540]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:54:59.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:59 smithi173 ceph-mon[57540]: osdmap e122: 8 total, 7 up, 8 in 2024-06-21T01:54:59.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:59 smithi173 ceph-mon[57540]: pgmap v103: 161 pgs: 14 peering, 21 stale+active+clean, 126 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-06-21T01:54:59.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:54:59 smithi173 ceph-mon[57540]: osdmap e123: 8 total, 7 up, 8 in 2024-06-21T01:54:59.993 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:59 smithi092 podman[77068]: 2024-06-21 01:54:59.60919811 +0000 UTC m=+0.017446396 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:54:59.993 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:59 smithi092 podman[77068]: 2024-06-21 01:54:59.711420653 +0000 UTC m=+0.119668978 container create 7e11805430ccbfa0439fc726055d096cc5e161cc9cdd8577e327408f992e77f4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-release-c65be81, ceph=True, io.buildah.version=1.36.0) 2024-06-21T01:54:59.993 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:59 smithi092 podman[77068]: 2024-06-21 01:54:59.922184547 +0000 UTC m=+0.330432840 container init 7e11805430ccbfa0439fc726055d096cc5e161cc9cdd8577e327408f992e77f4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-21T01:54:59.993 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:59 smithi092 podman[77068]: 2024-06-21 01:54:59.926098282 +0000 UTC m=+0.334346562 container start 7e11805430ccbfa0439fc726055d096cc5e161cc9cdd8577e327408f992e77f4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-06-21T01:54:59.994 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:54:59 smithi092 podman[77068]: 2024-06-21 01:54:59.997157995 +0000 UTC m=+0.405406274 container attach 7e11805430ccbfa0439fc726055d096cc5e161cc9cdd8577e327408f992e77f4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-06-21T01:55:00.351 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:00 smithi092 podman[77068]: 2024-06-21 01:55:00.179555972 +0000 UTC m=+0.587804261 container died 7e11805430ccbfa0439fc726055d096cc5e161cc9cdd8577e327408f992e77f4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-06-21T01:55:00.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:00 smithi173 ceph-mon[57540]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-06-21T01:55:00.765 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:00 smithi092 ceph-mon[64276]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-06-21T01:55:00.765 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:00 smithi092 ceph-mon[66756]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-06-21T01:55:01.101 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:00 smithi092 podman[77092]: 2024-06-21 01:55:00.767214618 +0000 UTC m=+0.581769070 container remove 7e11805430ccbfa0439fc726055d096cc5e161cc9cdd8577e327408f992e77f4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2) 2024-06-21T01:55:01.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:00 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.3.service: Deactivated successfully. 2024-06-21T01:55:01.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:00 smithi092 systemd[1]: Stopped Ceph osd.3 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:55:01.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:00 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.3.service: Consumed 8.464s CPU time. 2024-06-21T01:55:01.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:00 smithi092 systemd[1]: Starting Ceph osd.3 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:55:01.500 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:01 smithi092 ceph-mon[64276]: pgmap v105: 161 pgs: 20 peering, 20 stale+active+clean, 121 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-06-21T01:55:01.500 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:01 smithi092 ceph-mon[66756]: pgmap v105: 161 pgs: 20 peering, 20 stale+active+clean, 121 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-06-21T01:55:01.500 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 podman[77189]: 2024-06-21 01:55:01.20989141 +0000 UTC m=+0.113630172 container create 1db3cc61f03ecc29a56cdc9e4510cb697da761c25dc7971a9027dd105508fdf9 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-21T01:55:01.500 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 podman[77189]: 2024-06-21 01:55:01.113951753 +0000 UTC m=+0.017690488 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:55:01.500 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 podman[77189]: 2024-06-21 01:55:01.445884216 +0000 UTC m=+0.349622949 container init 1db3cc61f03ecc29a56cdc9e4510cb697da761c25dc7971a9027dd105508fdf9 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-06-21T01:55:01.500 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 podman[77189]: 2024-06-21 01:55:01.44952989 +0000 UTC m=+0.353268613 container start 1db3cc61f03ecc29a56cdc9e4510cb697da761c25dc7971a9027dd105508fdf9 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, ceph=True, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2) 2024-06-21T01:55:01.501 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 podman[77189]: 2024-06-21 01:55:01.503969171 +0000 UTC m=+0.407707895 container attach 1db3cc61f03ecc29a56cdc9e4510cb697da761c25dc7971a9027dd105508fdf9 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-21T01:55:01.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:01 smithi173 ceph-mon[57540]: pgmap v105: 161 pgs: 20 peering, 20 stale+active+clean, 121 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-06-21T01:55:02.101 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate[77203]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-21T01:55:02.101 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 bash[77189]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-21T01:55:02.101 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate[77203]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-06-21T01:55:02.101 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 bash[77189]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-06-21T01:55:02.101 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate[77203]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-06-21T01:55:02.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 bash[77189]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-06-21T01:55:02.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate[77203]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-06-21T01:55:02.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 bash[77189]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-06-21T01:55:02.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate[77203]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-06-21T01:55:02.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 bash[77189]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-06-21T01:55:02.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate[77203]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-21T01:55:02.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 bash[77189]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-21T01:55:02.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate[77203]: --> ceph-volume raw activate successful for osd ID: 3 2024-06-21T01:55:02.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 bash[77189]: --> ceph-volume raw activate successful for osd ID: 3 2024-06-21T01:55:02.102 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:01 smithi092 podman[77189]: 2024-06-21 01:55:01.869902199 +0000 UTC m=+0.773640919 container died 1db3cc61f03ecc29a56cdc9e4510cb697da761c25dc7971a9027dd105508fdf9 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-06-21T01:55:02.601 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:02 smithi092 podman[77336]: 2024-06-21 01:55:02.457700198 +0000 UTC m=+0.581911316 container remove 1db3cc61f03ecc29a56cdc9e4510cb697da761c25dc7971a9027dd105508fdf9 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81) 2024-06-21T01:55:02.933 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:02 smithi092 podman[77395]: 2024-06-21 01:55:02.577621682 +0000 UTC m=+0.017449847 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:55:02.933 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:02 smithi092 podman[77395]: 2024-06-21 01:55:02.700251087 +0000 UTC m=+0.140079241 container create f0817e4dfec7771d4d21b5a9347fc8005b10e886368dbf0e812685e4ec7a418f (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-06-21T01:55:02.933 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:02 smithi092 podman[77395]: 2024-06-21 01:55:02.9113444 +0000 UTC m=+0.351172570 container init f0817e4dfec7771d4d21b5a9347fc8005b10e886368dbf0e812685e4ec7a418f (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-06-21T01:55:02.933 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:02 smithi092 podman[77395]: 2024-06-21 01:55:02.915503306 +0000 UTC m=+0.355331461 container start f0817e4dfec7771d4d21b5a9347fc8005b10e886368dbf0e812685e4ec7a418f (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240618, ceph=True, CEPH_POINT_RELEASE=) 2024-06-21T01:55:03.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:02 smithi092 ceph-mon[64276]: pgmap v106: 161 pgs: 27 active+undersized, 20 peering, 19 active+undersized+degraded, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 67/723 objects degraded (9.267%) 2024-06-21T01:55:03.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:02 smithi092 ceph-mon[64276]: Health check failed: Degraded data redundancy: 67/723 objects degraded (9.267%), 19 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:03.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:02 smithi092 ceph-mon[66756]: pgmap v106: 161 pgs: 27 active+undersized, 20 peering, 19 active+undersized+degraded, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 67/723 objects degraded (9.267%) 2024-06-21T01:55:03.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:02 smithi092 ceph-mon[66756]: Health check failed: Degraded data redundancy: 67/723 objects degraded (9.267%), 19 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:03.353 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:02 smithi092 ceph-osd[77414]: -- 172.21.15.92:0/1104256444 <== mon.1 v2:172.21.15.92:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e423423a40 con 0x55e4241e2000 2024-06-21T01:55:03.353 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:03 smithi092 bash[77395]: f0817e4dfec7771d4d21b5a9347fc8005b10e886368dbf0e812685e4ec7a418f 2024-06-21T01:55:03.353 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:03 smithi092 systemd[1]: Started Ceph osd.3 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:55:03.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:02 smithi173 ceph-mon[57540]: pgmap v106: 161 pgs: 27 active+undersized, 20 peering, 19 active+undersized+degraded, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 67/723 objects degraded (9.267%) 2024-06-21T01:55:03.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:02 smithi173 ceph-mon[57540]: Health check failed: Degraded data redundancy: 67/723 objects degraded (9.267%), 19 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:03.777 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:03 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[77410]: 2024-06-21T01:55:03.466+0000 7fb9576b8640 -1 Falling back to public interface 2024-06-21T01:55:03.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:04.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:04 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:04.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:04 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:04.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:04 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:04.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:04 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:04.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:04 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:04.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:04 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:04.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:04 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:04.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:04 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:04.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:04 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:05 smithi092 ceph-mon[64276]: Health check update: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-21T01:55:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:05 smithi092 ceph-mon[64276]: pgmap v107: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-06-21T01:55:05.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:05 smithi092 ceph-mon[66756]: Health check update: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-21T01:55:05.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:05 smithi092 ceph-mon[66756]: pgmap v107: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-06-21T01:55:05.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:05 smithi173 ceph-mon[57540]: Health check update: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-06-21T01:55:05.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:05 smithi173 ceph-mon[57540]: pgmap v107: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-06-21T01:55:06.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:06 smithi092 ceph-mon[64276]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-06-21T01:55:06.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:06 smithi092 ceph-mon[66756]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-06-21T01:55:06.351 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:06 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[77410]: 2024-06-21T01:55:06.182+0000 7fb9576b8640 -1 osd.3 0 read_superblock omap replica is missing. 2024-06-21T01:55:06.352 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:06 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[77410]: 2024-06-21T01:55:06.211+0000 7fb9576b8640 -1 osd.3 121 log_to_monitors true 2024-06-21T01:55:06.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:06 smithi173 ceph-mon[57540]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-06-21T01:55:07.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:07 smithi092 ceph-mon[64276]: from='osd.3 [v2:172.21.15.92:6826/1237137747,v1:172.21.15.92:6827/1237137747]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:55:07.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:07 smithi092 ceph-mon[64276]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:55:07.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:07 smithi092 ceph-mon[64276]: pgmap v108: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-06-21T01:55:07.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:07 smithi092 ceph-mon[66756]: from='osd.3 [v2:172.21.15.92:6826/1237137747,v1:172.21.15.92:6827/1237137747]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:55:07.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:07 smithi092 ceph-mon[66756]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:55:07.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:07 smithi092 ceph-mon[66756]: pgmap v108: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-06-21T01:55:07.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:07 smithi173 ceph-mon[57540]: from='osd.3 [v2:172.21.15.92:6826/1237137747,v1:172.21.15.92:6827/1237137747]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:55:07.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:07 smithi173 ceph-mon[57540]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-21T01:55:07.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:07 smithi173 ceph-mon[57540]: pgmap v108: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-06-21T01:55:07.851 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 01:55:07 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[77410]: 2024-06-21T01:55:07.554+0000 7fb94b45b640 -1 osd.3 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:55:08.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[66756]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-21T01:55:08.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[66756]: osdmap e124: 8 total, 7 up, 8 in 2024-06-21T01:55:08.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[66756]: from='osd.3 [v2:172.21.15.92:6826/1237137747,v1:172.21.15.92:6827/1237137747]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:55:08.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[66756]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:55:08.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[66756]: from='osd.3 ' entity='osd.3' 2024-06-21T01:55:08.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[64276]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-21T01:55:08.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[64276]: osdmap e124: 8 total, 7 up, 8 in 2024-06-21T01:55:08.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[64276]: from='osd.3 [v2:172.21.15.92:6826/1237137747,v1:172.21.15.92:6827/1237137747]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:55:08.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[64276]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:55:08.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[64276]: from='osd.3 ' entity='osd.3' 2024-06-21T01:55:08.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:08 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:08 smithi173 ceph-mon[57540]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-21T01:55:08.504 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:08 smithi173 ceph-mon[57540]: osdmap e124: 8 total, 7 up, 8 in 2024-06-21T01:55:08.504 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:08 smithi173 ceph-mon[57540]: from='osd.3 [v2:172.21.15.92:6826/1237137747,v1:172.21.15.92:6827/1237137747]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:55:08.504 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:08 smithi173 ceph-mon[57540]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-06-21T01:55:08.504 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:08 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.504 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:08 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.504 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:08 smithi173 ceph-mon[57540]: from='osd.3 ' entity='osd.3' 2024-06-21T01:55:08.504 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:08 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.504 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:08 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:08.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:09.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:55:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:55:09] "GET /metrics HTTP/1.1" 200 38428 "" "Prometheus/2.33.4" 2024-06-21T01:55:09.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:09 smithi092 ceph-mon[64276]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:55:09.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:09 smithi092 ceph-mon[64276]: osd.3 [v2:172.21.15.92:6826/1237137747,v1:172.21.15.92:6827/1237137747] boot 2024-06-21T01:55:09.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:09 smithi092 ceph-mon[64276]: osdmap e125: 8 total, 8 up, 8 in 2024-06-21T01:55:09.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:09 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:55:09.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:09 smithi092 ceph-mon[64276]: pgmap v111: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-06-21T01:55:09.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:09 smithi092 ceph-mon[66756]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:55:09.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:09 smithi092 ceph-mon[66756]: osd.3 [v2:172.21.15.92:6826/1237137747,v1:172.21.15.92:6827/1237137747] boot 2024-06-21T01:55:09.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:09 smithi092 ceph-mon[66756]: osdmap e125: 8 total, 8 up, 8 in 2024-06-21T01:55:09.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:09 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:55:09.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:09 smithi092 ceph-mon[66756]: pgmap v111: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-06-21T01:55:09.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:09 smithi173 ceph-mon[57540]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:55:09.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:09 smithi173 ceph-mon[57540]: osd.3 [v2:172.21.15.92:6826/1237137747,v1:172.21.15.92:6827/1237137747] boot 2024-06-21T01:55:09.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:09 smithi173 ceph-mon[57540]: osdmap e125: 8 total, 8 up, 8 in 2024-06-21T01:55:09.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:09 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-21T01:55:09.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:09 smithi173 ceph-mon[57540]: pgmap v111: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-06-21T01:55:10.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:10 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 95/723 objects degraded (13.140%), 26 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:10.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:10 smithi092 ceph-mon[66756]: osdmap e126: 8 total, 8 up, 8 in 2024-06-21T01:55:10.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:10 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 95/723 objects degraded (13.140%), 26 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:10.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:10 smithi092 ceph-mon[64276]: osdmap e126: 8 total, 8 up, 8 in 2024-06-21T01:55:10.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:10 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 95/723 objects degraded (13.140%), 26 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:10.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:10 smithi173 ceph-mon[57540]: osdmap e126: 8 total, 8 up, 8 in 2024-06-21T01:55:11.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:55:11.244+0000 7f9965fe7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-21T01:55:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: pgmap v113: 161 pgs: 5 peering, 36 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/723 objects degraded (13.001%) 2024-06-21T01:55:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:55:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:55:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:55:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: pgmap v113: 161 pgs: 5 peering, 36 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/723 objects degraded (13.001%) 2024-06-21T01:55:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:55:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:55:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:55:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:12.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: pgmap v113: 161 pgs: 5 peering, 36 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/723 objects degraded (13.001%) 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:12.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:13.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:12 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:13.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:12 smithi173 ceph-mon[57540]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-21T01:55:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:12 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:12 smithi092 ceph-mon[64276]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-21T01:55:13.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:12 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:13.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:12 smithi092 ceph-mon[66756]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-21T01:55:13.935 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:13 smithi092 ceph-mon[64276]: pgmap v114: 161 pgs: 5 peering, 5 active+undersized, 3 active+undersized+degraded, 148 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 16/723 objects degraded (2.213%) 2024-06-21T01:55:14.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:13 smithi092 ceph-mon[66756]: pgmap v114: 161 pgs: 5 peering, 5 active+undersized, 3 active+undersized+degraded, 148 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 16/723 objects degraded (2.213%) 2024-06-21T01:55:14.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:13 smithi173 ceph-mon[57540]: pgmap v114: 161 pgs: 5 peering, 5 active+undersized, 3 active+undersized+degraded, 148 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 16/723 objects degraded (2.213%) 2024-06-21T01:55:15.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:14 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:15.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:14 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 16/723 objects degraded (2.213%), 3 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:15.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:14 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:15.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:14 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 16/723 objects degraded (2.213%), 3 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:15.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:14 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:15.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:14 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 16/723 objects degraded (2.213%), 3 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:15 smithi173 ceph-mon[57540]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:15 smithi173 ceph-mon[57540]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/723 objects degraded (2.213%), 3 pgs degraded) 2024-06-21T01:55:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:15 smithi173 ceph-mon[57540]: Cluster is now healthy 2024-06-21T01:55:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:15 smithi092 ceph-mon[64276]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:15 smithi092 ceph-mon[64276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/723 objects degraded (2.213%), 3 pgs degraded) 2024-06-21T01:55:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:15 smithi092 ceph-mon[64276]: Cluster is now healthy 2024-06-21T01:55:16.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:15 smithi092 ceph-mon[66756]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:16.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:15 smithi092 ceph-mon[66756]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/723 objects degraded (2.213%), 3 pgs degraded) 2024-06-21T01:55:16.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:15 smithi092 ceph-mon[66756]: Cluster is now healthy 2024-06-21T01:55:17.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:16 smithi173 ceph-mon[57540]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:17.228 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:16 smithi092 ceph-mon[64276]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:17.229 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:16 smithi092 ceph-mon[66756]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:18.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:19.265 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:18 smithi092 ceph-mon[64276]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:19.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:18 smithi092 ceph-mon[66756]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:19.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:18 smithi173 ceph-mon[57540]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:19.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:55:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:55:19] "GET /metrics HTTP/1.1" 200 38225 "" "Prometheus/2.33.4" 2024-06-21T01:55:21.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:20 smithi173 ceph-mon[57540]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:21.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:20 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:20 smithi092 ceph-mon[64276]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:20 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:21.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:20 smithi092 ceph-mon[66756]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:55:21.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:20 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:23.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:22 smithi092 ceph-mon[64276]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-06-21T01:55:23.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:22 smithi092 ceph-mon[66756]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-06-21T01:55:23.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:22 smithi173 ceph-mon[57540]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-06-21T01:55:23.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:23 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:24.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:24 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:24.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:24 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:24.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:24 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:25.218 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:55:25.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:25 smithi092 ceph-mon[64276]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:25.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:25 smithi092 ceph-mon[66756]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:25.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:25 smithi173 ceph-mon[57540]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:25.871 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (4m) 18s ago 8m 21.0M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (4m) 2m ago 8m 46.8M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (3m) 18s ago 7m 52.6M - 3.5 e1d6a67b021e 3192bc59456b 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (3m) 2m ago 13m 438M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (4m) 18s ago 14m 502M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (3m) 18s ago 15m 49.7M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (2m) 2m ago 13m 22.2M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (3m) 18s ago 13m 42.0M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (4m) 18s ago 9m 16.8M - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (4m) 2m ago 9m 20.3M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (2m) 18s ago 13m 52.6M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 7c252b0f2dfd 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (88s) 18s ago 12m 44.8M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 9ad8cda61390 2024-06-21T01:55:25.872 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (55s) 18s ago 12m 49.8M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 0fe024e7dad9 2024-06-21T01:55:25.873 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (23s) 18s ago 11m 14.5M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 f0817e4dfec7 2024-06-21T01:55:25.873 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (11m) 2m ago 11m 56.0M 2992M 17.2.0 e1d6a67b021e 7cbd617767bd 2024-06-21T01:55:25.873 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (11m) 2m ago 11m 54.0M 2992M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:55:25.873 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (10m) 2m ago 10m 48.9M 2992M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:55:25.873 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (10m) 2m ago 10m 53.2M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:55:25.873 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (3m) 2m ago 9m 60.0M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:55:25.873 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (8m) 18s ago 8m 76.8M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:55:25.873 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (8m) 2m ago 8m 72.6M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:55:25.873 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (8m) 18s ago 8m 74.5M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:55:25.873 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (7m) 2m ago 7m 73.8M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:55:26.267 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:55:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:55:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:55:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:55:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:55:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:55:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:55:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:55:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-06-21T01:55:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 4 2024-06-21T01:55:26.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:55:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:55:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:55:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:55:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:55:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-06-21T01:55:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 9 2024-06-21T01:55:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:55:26.268 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:55:26.595 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:55:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:55:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:55:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:55:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:55:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr", 2024-06-21T01:55:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "mon" 2024-06-21T01:55:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:55:26.595 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "9/23 daemons upgraded", 2024-06-21T01:55:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading osd daemons", 2024-06-21T01:55:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:55:26.596 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:55:26.992 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:55:27.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:26 smithi173 ceph-mon[57540]: from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:27.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:26 smithi173 ceph-mon[57540]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:27.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:26 smithi173 ceph-mon[57540]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:27.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:27.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:27.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:27.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:26 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/843336484' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:27.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[66756]: from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:27.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[66756]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:27.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[66756]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:27.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:27.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:27.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:27.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/843336484' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:27.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[64276]: from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:27.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[64276]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:27.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[64276]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:27.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:27.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:27.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:27.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:26 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/843336484' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:27 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:27 smithi173 ceph-mon[57540]: Upgrade: osd.4 is safe to restart 2024-06-21T01:55:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:27 smithi173 ceph-mon[57540]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:27 smithi173 ceph-mon[57540]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:27 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/2959752635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:55:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:27 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:27 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-21T01:55:28.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:27 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:55:28.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:28.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[66756]: Upgrade: osd.4 is safe to restart 2024-06-21T01:55:28.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[66756]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:28.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[66756]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:28.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/2959752635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:55:28.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:28.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-21T01:55:28.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:55:28.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-21T01:55:28.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[64276]: Upgrade: osd.4 is safe to restart 2024-06-21T01:55:28.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[64276]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:28.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[64276]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:28.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/2959752635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:55:28.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:28.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-21T01:55:28.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:27 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:55:28.883 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:28 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:29.185 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:28 smithi173 ceph-mon[57540]: Upgrade: Updating osd.4 2024-06-21T01:55:29.185 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:28 smithi173 ceph-mon[57540]: Deploying daemon osd.4 on smithi173 2024-06-21T01:55:29.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:28 smithi092 ceph-mon[64276]: Upgrade: Updating osd.4 2024-06-21T01:55:29.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:28 smithi092 ceph-mon[64276]: Deploying daemon osd.4 on smithi173 2024-06-21T01:55:29.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:28 smithi092 ceph-mon[66756]: Upgrade: Updating osd.4 2024-06-21T01:55:29.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:28 smithi092 ceph-mon[66756]: Deploying daemon osd.4 on smithi173 2024-06-21T01:55:29.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:55:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:55:29] "GET /metrics HTTP/1.1" 200 38432 "" "Prometheus/2.33.4" 2024-06-21T01:55:30.125 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:29 smithi173 ceph-mon[57540]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:55:30.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:29 smithi092 ceph-mon[64276]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:55:30.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:29 smithi092 ceph-mon[66756]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:55:30.728 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:30 smithi173 systemd[1]: Stopping Ceph osd.4 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:55:31.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:30 smithi173 ceph-mon[57540]: osd.4 marked itself down and dead 2024-06-21T01:55:31.228 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:30 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[29372]: 2024-06-21T01:55:30.733+0000 7f326a411700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:55:31.229 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:30 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[29372]: 2024-06-21T01:55:30.733+0000 7f326a411700 -1 osd.4 126 *** Got signal Terminated *** 2024-06-21T01:55:31.229 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:30 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[29372]: 2024-06-21T01:55:30.733+0000 7f326a411700 -1 osd.4 126 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T01:55:31.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:30 smithi092 ceph-mon[64276]: osd.4 marked itself down and dead 2024-06-21T01:55:31.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:30 smithi092 ceph-mon[66756]: osd.4 marked itself down and dead 2024-06-21T01:55:31.647 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:31 smithi173 podman[60308]: 2024-06-21 01:55:31.374432718 +0000 UTC m=+0.739007212 container died 7cbd617767bd6473febf0d3fcf225f1b695a1720e8232e4e4f56a242378a8f12 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, version=8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, maintainer=Guillaume Abrioux , GIT_CLEAN=True, architecture=x86_64, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, distribution-scope=public, RELEASE=HEAD, com.redhat.component=centos-stream-container, name=centos-stream, build-date=2022-05-03T08:36:31.336870) 2024-06-21T01:55:31.944 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:31 smithi173 podman[60308]: 2024-06-21 01:55:31.73391337 +0000 UTC m=+1.098487895 container cleanup 7cbd617767bd6473febf0d3fcf225f1b695a1720e8232e4e4f56a242378a8f12 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4, com.redhat.component=centos-stream-container, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, RELEASE=HEAD, release=754, GIT_BRANCH=HEAD, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-06-21T01:55:31.944 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:31 smithi173 bash[60308]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4 2024-06-21T01:55:32.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:31 smithi173 ceph-mon[57540]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:32.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:31 smithi173 ceph-mon[57540]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:55:32.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:31 smithi173 ceph-mon[57540]: osdmap e127: 8 total, 7 up, 8 in 2024-06-21T01:55:32.228 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:32 smithi173 podman[60335]: 2024-06-21 01:55:32.060747816 +0000 UTC m=+0.683093443 container remove 7cbd617767bd6473febf0d3fcf225f1b695a1720e8232e4e4f56a242378a8f12 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_CLEAN=True, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc.) 2024-06-21T01:55:32.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:31 smithi092 ceph-mon[64276]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:32.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:31 smithi092 ceph-mon[64276]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:55:32.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:31 smithi092 ceph-mon[64276]: osdmap e127: 8 total, 7 up, 8 in 2024-06-21T01:55:32.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:31 smithi092 ceph-mon[66756]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:32.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:31 smithi092 ceph-mon[66756]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:55:32.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:31 smithi092 ceph-mon[66756]: osdmap e127: 8 total, 7 up, 8 in 2024-06-21T01:55:32.692 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:32 smithi173 podman[60425]: 2024-06-21 01:55:32.305306348 +0000 UTC m=+0.017405996 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:55:32.692 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:32 smithi173 podman[60425]: 2024-06-21 01:55:32.403675904 +0000 UTC m=+0.115775547 container create 5b1712141719530a205d11736d534c331e6e754cd44316c2046d5ebd2c122ee5 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-deactivate, RELEASE=squid-release-c65be81, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-06-21T01:55:32.951 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:32 smithi173 podman[60425]: 2024-06-21 01:55:32.696645818 +0000 UTC m=+0.408745464 container init 5b1712141719530a205d11736d534c331e6e754cd44316c2046d5ebd2c122ee5 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-06-21T01:55:32.951 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:32 smithi173 podman[60425]: 2024-06-21 01:55:32.70087603 +0000 UTC m=+0.412975684 container start 5b1712141719530a205d11736d534c331e6e754cd44316c2046d5ebd2c122ee5 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-06-21T01:55:32.951 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:32 smithi173 podman[60425]: 2024-06-21 01:55:32.789928663 +0000 UTC m=+0.502028305 container attach 5b1712141719530a205d11736d534c331e6e754cd44316c2046d5ebd2c122ee5 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-21T01:55:33.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:32 smithi173 ceph-mon[57540]: osdmap e128: 8 total, 7 up, 8 in 2024-06-21T01:55:33.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:32 smithi173 ceph-mon[57540]: pgmap v126: 161 pgs: 8 active+undersized, 43 peering, 5 stale+active+clean, 2 active+undersized+degraded, 103 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 6/723 objects degraded (0.830%) 2024-06-21T01:55:33.228 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:32 smithi173 podman[60425]: 2024-06-21 01:55:32.955020416 +0000 UTC m=+0.667120071 container died 5b1712141719530a205d11736d534c331e6e754cd44316c2046d5ebd2c122ee5 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T01:55:33.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:32 smithi092 ceph-mon[66756]: osdmap e128: 8 total, 7 up, 8 in 2024-06-21T01:55:33.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:32 smithi092 ceph-mon[66756]: pgmap v126: 161 pgs: 8 active+undersized, 43 peering, 5 stale+active+clean, 2 active+undersized+degraded, 103 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 6/723 objects degraded (0.830%) 2024-06-21T01:55:33.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:32 smithi092 ceph-mon[64276]: osdmap e128: 8 total, 7 up, 8 in 2024-06-21T01:55:33.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:32 smithi092 ceph-mon[64276]: pgmap v126: 161 pgs: 8 active+undersized, 43 peering, 5 stale+active+clean, 2 active+undersized+degraded, 103 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 6/723 objects degraded (0.830%) 2024-06-21T01:55:33.728 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:33 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:33.728 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:33 smithi173 podman[60449]: 2024-06-21 01:55:33.585941834 +0000 UTC m=+0.620285330 container remove 5b1712141719530a205d11736d534c331e6e754cd44316c2046d5ebd2c122ee5 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-deactivate, RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T01:55:33.729 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:33 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.4.service: Deactivated successfully. 2024-06-21T01:55:33.729 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:33 smithi173 systemd[1]: Stopped Ceph osd.4 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:55:33.729 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:33 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.4.service: Consumed 8.346s CPU time. 2024-06-21T01:55:34.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:33 smithi092 ceph-mon[64276]: Health check failed: Reduced data availability: 6 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-06-21T01:55:34.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:33 smithi092 ceph-mon[64276]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 2 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:34.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:33 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:33 smithi092 ceph-mon[66756]: Health check failed: Reduced data availability: 6 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-06-21T01:55:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:33 smithi092 ceph-mon[66756]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 2 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:34.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:33 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:34.163 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:33 smithi173 ceph-mon[57540]: Health check failed: Reduced data availability: 6 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-06-21T01:55:34.163 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:33 smithi173 ceph-mon[57540]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 2 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:34.163 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:33 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:34.163 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:33 smithi173 systemd[1]: Starting Ceph osd.4 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:55:34.163 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 podman[60549]: 2024-06-21 01:55:33.923958495 +0000 UTC m=+0.017530659 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:55:34.163 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 podman[60549]: 2024-06-21 01:55:34.054701635 +0000 UTC m=+0.148273858 container create 2c1a2629d048f333f02da4bbb0804a1eafe14edd3389206ff9628d6704666ca7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux ) 2024-06-21T01:55:34.478 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 podman[60549]: 2024-06-21 01:55:34.307374303 +0000 UTC m=+0.400946467 container init 2c1a2629d048f333f02da4bbb0804a1eafe14edd3389206ff9628d6704666ca7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, GIT_CLEAN=True, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-06-21T01:55:34.479 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 podman[60549]: 2024-06-21 01:55:34.310798476 +0000 UTC m=+0.404370646 container start 2c1a2629d048f333f02da4bbb0804a1eafe14edd3389206ff9628d6704666ca7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0) 2024-06-21T01:55:34.479 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 podman[60549]: 2024-06-21 01:55:34.382389381 +0000 UTC m=+0.475961549 container attach 2c1a2629d048f333f02da4bbb0804a1eafe14edd3389206ff9628d6704666ca7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, GIT_CLEAN=True) 2024-06-21T01:55:34.966 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate[60565]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-21T01:55:34.966 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 bash[60549]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-21T01:55:34.966 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate[60565]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-21T01:55:34.966 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 bash[60549]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-21T01:55:34.966 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate[60565]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-21T01:55:34.966 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 bash[60549]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-21T01:55:34.966 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate[60565]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-21T01:55:34.966 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 bash[60549]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-21T01:55:34.966 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate[60565]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-06-21T01:55:34.966 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 bash[60549]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-06-21T01:55:34.966 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate[60565]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-21T01:55:34.967 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 bash[60549]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-21T01:55:34.967 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate[60565]: --> ceph-volume raw activate successful for osd ID: 4 2024-06-21T01:55:34.967 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 bash[60549]: --> ceph-volume raw activate successful for osd ID: 4 2024-06-21T01:55:34.967 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:34 smithi173 podman[60549]: 2024-06-21 01:55:34.727045696 +0000 UTC m=+0.820617867 container died 2c1a2629d048f333f02da4bbb0804a1eafe14edd3389206ff9628d6704666ca7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-21T01:55:35.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:34 smithi173 ceph-mon[57540]: pgmap v127: 161 pgs: 16 active+undersized, 43 peering, 9 active+undersized+degraded, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 41/723 objects degraded (5.671%) 2024-06-21T01:55:35.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:34 smithi092 ceph-mon[64276]: pgmap v127: 161 pgs: 16 active+undersized, 43 peering, 9 active+undersized+degraded, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 41/723 objects degraded (5.671%) 2024-06-21T01:55:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:34 smithi092 ceph-mon[66756]: pgmap v127: 161 pgs: 16 active+undersized, 43 peering, 9 active+undersized+degraded, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 41/723 objects degraded (5.671%) 2024-06-21T01:55:35.691 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:35 smithi173 podman[60684]: 2024-06-21 01:55:35.362044726 +0000 UTC m=+0.626295680 container remove 2c1a2629d048f333f02da4bbb0804a1eafe14edd3389206ff9628d6704666ca7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-activate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True) 2024-06-21T01:55:35.691 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:35 smithi173 podman[60744]: 2024-06-21 01:55:35.479376836 +0000 UTC m=+0.017393055 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:55:35.691 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:35 smithi173 podman[60744]: 2024-06-21 01:55:35.588032621 +0000 UTC m=+0.126048831 container create d42a6ffd2e665eea2feee144895412d4a82a3f96e77d4b09cbb1d21366f1eedc (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-06-21T01:55:36.228 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:35 smithi173 podman[60744]: 2024-06-21 01:55:35.823814433 +0000 UTC m=+0.361830659 container init d42a6ffd2e665eea2feee144895412d4a82a3f96e77d4b09cbb1d21366f1eedc (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-06-21T01:55:36.228 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:35 smithi173 podman[60744]: 2024-06-21 01:55:35.827019791 +0000 UTC m=+0.365036011 container start d42a6ffd2e665eea2feee144895412d4a82a3f96e77d4b09cbb1d21366f1eedc (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-06-21T01:55:36.229 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:35 smithi173 ceph-osd[60763]: -- 172.21.15.173:0/2951191497 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561ac76ada40 con 0x561ac768bc00 2024-06-21T01:55:36.229 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:35 smithi173 bash[60744]: d42a6ffd2e665eea2feee144895412d4a82a3f96e77d4b09cbb1d21366f1eedc 2024-06-21T01:55:36.229 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:35 smithi173 systemd[1]: Started Ceph osd.4 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:55:36.654 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T01:55:36.372+0000 7f051d98b640 -1 Falling back to public interface 2024-06-21T01:55:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:36 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:36 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:36 smithi173 ceph-mon[57540]: pgmap v128: 161 pgs: 41 active+undersized, 27 active+undersized+degraded, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 126/723 objects degraded (17.427%) 2024-06-21T01:55:37.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:36 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:37.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:36 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:37.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:36 smithi092 ceph-mon[64276]: pgmap v128: 161 pgs: 41 active+undersized, 27 active+undersized+degraded, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 126/723 objects degraded (17.427%) 2024-06-21T01:55:37.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:36 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:37.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:36 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:37.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:36 smithi092 ceph-mon[66756]: pgmap v128: 161 pgs: 41 active+undersized, 27 active+undersized+degraded, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 126/723 objects degraded (17.427%) 2024-06-21T01:55:38.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:37 smithi173 ceph-mon[57540]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 12 pgs peering) 2024-06-21T01:55:38.299 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:37 smithi092 ceph-mon[64276]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 12 pgs peering) 2024-06-21T01:55:38.299 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:37 smithi092 ceph-mon[66756]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 12 pgs peering) 2024-06-21T01:55:38.966 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:38 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:39.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:38 smithi173 ceph-mon[57540]: pgmap v129: 161 pgs: 41 active+undersized, 27 active+undersized+degraded, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 126/723 objects degraded (17.427%) 2024-06-21T01:55:39.228 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:39 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T01:55:39.081+0000 7f051d98b640 -1 osd.4 0 read_superblock omap replica is missing. 2024-06-21T01:55:39.228 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:39 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T01:55:39.112+0000 7f051d98b640 -1 osd.4 126 log_to_monitors true 2024-06-21T01:55:39.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:38 smithi092 ceph-mon[64276]: pgmap v129: 161 pgs: 41 active+undersized, 27 active+undersized+degraded, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 126/723 objects degraded (17.427%) 2024-06-21T01:55:39.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:38 smithi092 ceph-mon[66756]: pgmap v129: 161 pgs: 41 active+undersized, 27 active+undersized+degraded, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 126/723 objects degraded (17.427%) 2024-06-21T01:55:39.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:55:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:55:39] "GET /metrics HTTP/1.1" 200 38432 "" "Prometheus/2.33.4" 2024-06-21T01:55:40.228 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 01:55:40 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T01:55:40.020+0000 7f0515736640 -1 osd.4 126 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:55:40.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:39 smithi173 ceph-mon[57540]: from='osd.4 [v2:172.21.15.173:6800/1418821372,v1:172.21.15.173:6801/1418821372]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:55:40.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:39 smithi173 ceph-mon[57540]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:55:40.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:39 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 126/723 objects degraded (17.427%), 27 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:40.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:40.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:39 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:40.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:39 smithi092 ceph-mon[64276]: from='osd.4 [v2:172.21.15.173:6800/1418821372,v1:172.21.15.173:6801/1418821372]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:55:40.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:39 smithi092 ceph-mon[64276]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:55:40.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:39 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 126/723 objects degraded (17.427%), 27 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:40.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:40.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:39 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:40.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:39 smithi092 ceph-mon[66756]: from='osd.4 [v2:172.21.15.173:6800/1418821372,v1:172.21.15.173:6801/1418821372]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:55:40.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:39 smithi092 ceph-mon[66756]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-21T01:55:40.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:39 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 126/723 objects degraded (17.427%), 27 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:40.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:40.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:39 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:41.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:41 smithi173 ceph-mon[57540]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-21T01:55:41.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:41 smithi173 ceph-mon[57540]: from='osd.4 [v2:172.21.15.173:6800/1418821372,v1:172.21.15.173:6801/1418821372]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:55:41.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:41 smithi173 ceph-mon[57540]: osdmap e129: 8 total, 7 up, 8 in 2024-06-21T01:55:41.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:41 smithi173 ceph-mon[57540]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:55:41.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:41.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:41.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:41 smithi173 ceph-mon[57540]: pgmap v131: 161 pgs: 41 active+undersized, 27 active+undersized+degraded, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 126/723 objects degraded (17.427%) 2024-06-21T01:55:41.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:41.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:41.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[64276]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-21T01:55:41.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[64276]: from='osd.4 [v2:172.21.15.173:6800/1418821372,v1:172.21.15.173:6801/1418821372]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:55:41.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[64276]: osdmap e129: 8 total, 7 up, 8 in 2024-06-21T01:55:41.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[64276]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:55:41.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:41.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:41.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[64276]: pgmap v131: 161 pgs: 41 active+undersized, 27 active+undersized+degraded, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 126/723 objects degraded (17.427%) 2024-06-21T01:55:41.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:41.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[66756]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-21T01:55:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[66756]: from='osd.4 [v2:172.21.15.173:6800/1418821372,v1:172.21.15.173:6801/1418821372]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:55:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[66756]: osdmap e129: 8 total, 7 up, 8 in 2024-06-21T01:55:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[66756]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:55:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[66756]: pgmap v131: 161 pgs: 41 active+undersized, 27 active+undersized+degraded, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 126/723 objects degraded (17.427%) 2024-06-21T01:55:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:41.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:42 smithi173 ceph-mon[57540]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:55:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:42 smithi173 ceph-mon[57540]: osd.4 [v2:172.21.15.173:6800/1418821372,v1:172.21.15.173:6801/1418821372] boot 2024-06-21T01:55:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:42 smithi173 ceph-mon[57540]: osdmap e130: 8 total, 8 up, 8 in 2024-06-21T01:55:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:42 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:55:42.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:42 smithi173 ceph-mon[57540]: osdmap e131: 8 total, 8 up, 8 in 2024-06-21T01:55:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:42 smithi092 ceph-mon[64276]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:55:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:42 smithi092 ceph-mon[64276]: osd.4 [v2:172.21.15.173:6800/1418821372,v1:172.21.15.173:6801/1418821372] boot 2024-06-21T01:55:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:42 smithi092 ceph-mon[64276]: osdmap e130: 8 total, 8 up, 8 in 2024-06-21T01:55:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:42 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:55:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:42 smithi092 ceph-mon[64276]: osdmap e131: 8 total, 8 up, 8 in 2024-06-21T01:55:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:42 smithi092 ceph-mon[66756]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:55:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:42 smithi092 ceph-mon[66756]: osd.4 [v2:172.21.15.173:6800/1418821372,v1:172.21.15.173:6801/1418821372] boot 2024-06-21T01:55:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:42 smithi092 ceph-mon[66756]: osdmap e130: 8 total, 8 up, 8 in 2024-06-21T01:55:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:42 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-21T01:55:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:42 smithi092 ceph-mon[66756]: osdmap e131: 8 total, 8 up, 8 in 2024-06-21T01:55:43.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:43 smithi092 ceph-mon[64276]: pgmap v134: 161 pgs: 23 peering, 25 active+undersized, 16 active+undersized+degraded, 97 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 65/723 objects degraded (8.990%) 2024-06-21T01:55:43.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:43 smithi092 ceph-mon[66756]: pgmap v134: 161 pgs: 23 peering, 25 active+undersized, 16 active+undersized+degraded, 97 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 65/723 objects degraded (8.990%) 2024-06-21T01:55:43.364 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:43 smithi173 ceph-mon[57540]: pgmap v134: 161 pgs: 23 peering, 25 active+undersized, 16 active+undersized+degraded, 97 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 65/723 objects degraded (8.990%) 2024-06-21T01:55:43.665 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:43 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:44.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:55:43 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:55:43.844+0000 7f9965fe7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-06-21T01:55:44.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:44.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:44.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:55:44.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:55:44.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:44.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:55:44.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:44.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:44.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:44.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:55:44.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:55:44.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-06-21T01:55:44.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:44.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:44 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 65/723 objects degraded (8.990%), 16 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:45.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:45.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:45.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:55:45.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:55:45.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:45.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:55:45.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:45.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:45.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:45.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:55:45.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:55:45.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-06-21T01:55:45.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:45.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 65/723 objects degraded (8.990%), 16 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:45.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:45.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:45.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:55:45.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:55:45.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:45.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:55:45.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:45.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:45.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:45.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:55:45.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:55:45.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-06-21T01:55:45.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:45.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:44 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 65/723 objects degraded (8.990%), 16 pgs degraded (PG_DEGRADED) 2024-06-21T01:55:45.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:45 smithi173 ceph-mon[57540]: pgmap v135: 161 pgs: 23 peering, 18 active+undersized, 13 active+undersized+degraded, 107 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 48/723 objects degraded (6.639%) 2024-06-21T01:55:46.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:45 smithi092 ceph-mon[64276]: pgmap v135: 161 pgs: 23 peering, 18 active+undersized, 13 active+undersized+degraded, 107 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 48/723 objects degraded (6.639%) 2024-06-21T01:55:46.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:45 smithi092 ceph-mon[66756]: pgmap v135: 161 pgs: 23 peering, 18 active+undersized, 13 active+undersized+degraded, 107 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 48/723 objects degraded (6.639%) 2024-06-21T01:55:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:47 smithi092 ceph-mon[64276]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-06-21T01:55:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:47 smithi092 ceph-mon[64276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 48/723 objects degraded (6.639%), 13 pgs degraded) 2024-06-21T01:55:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:47 smithi092 ceph-mon[64276]: Cluster is now healthy 2024-06-21T01:55:47.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:47 smithi092 ceph-mon[66756]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-06-21T01:55:47.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:47 smithi092 ceph-mon[66756]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 48/723 objects degraded (6.639%), 13 pgs degraded) 2024-06-21T01:55:47.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:47 smithi092 ceph-mon[66756]: Cluster is now healthy 2024-06-21T01:55:47.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:47 smithi173 ceph-mon[57540]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-06-21T01:55:47.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:47 smithi173 ceph-mon[57540]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 48/723 objects degraded (6.639%), 13 pgs degraded) 2024-06-21T01:55:47.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:47 smithi173 ceph-mon[57540]: Cluster is now healthy 2024-06-21T01:55:48.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:48 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:49.265 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:49 smithi092 ceph-mon[64276]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:55:49.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:49 smithi092 ceph-mon[66756]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:55:49.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:49 smithi173 ceph-mon[57540]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:55:49.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:55:49 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:55:49] "GET /metrics HTTP/1.1" 200 38292 "" "Prometheus/2.33.4" 2024-06-21T01:55:51.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:51 smithi173 ceph-mon[57540]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-21T01:55:51.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:51 smithi092 ceph-mon[64276]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-21T01:55:51.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:51 smithi092 ceph-mon[66756]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-21T01:55:53.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:53 smithi092 ceph-mon[64276]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:55:53.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:53 smithi092 ceph-mon[66756]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:55:53.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:53 smithi173 ceph-mon[57540]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:55:53.978 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi092\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", domainname=\"(none)\", instance=\"smithi092\", job=\"node\", machine=\"x86_64\", nodename=\"smithi092\", release=\"5.14.0-460.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Jun 6 12:49:42 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:54.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:54 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:54.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:54 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:54.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:54 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:55:55.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:55 smithi092 ceph-mon[64276]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:55.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:55 smithi092 ceph-mon[66756]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:55.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:55 smithi173 ceph-mon[57540]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:57.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:57.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:57.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:57.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:57.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:55:57.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:55:57.352 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:55:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:55:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (4m) 50s ago 9m 21.0M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:55:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (4m) 18s ago 8m 55.6M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:55:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (4m) 50s ago 8m 52.6M - 3.5 e1d6a67b021e 3192bc59456b 2024-06-21T01:55:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (4m) 18s ago 13m 440M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:55:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (4m) 50s ago 15m 502M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:55:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (3m) 50s ago 15m 49.7M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:55:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (3m) 18s ago 14m 38.3M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:55:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (3m) 50s ago 14m 42.0M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:55:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (5m) 50s ago 10m 16.8M - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:55:57.998 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (4m) 18s ago 10m 20.7M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:55:57.999 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (2m) 50s ago 13m 52.6M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 7c252b0f2dfd 2024-06-21T01:55:57.999 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (2m) 50s ago 13m 44.8M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 9ad8cda61390 2024-06-21T01:55:57.999 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (87s) 50s ago 12m 49.8M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 0fe024e7dad9 2024-06-21T01:55:57.999 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (55s) 50s ago 12m 14.5M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 f0817e4dfec7 2024-06-21T01:55:57.999 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (22s) 18s ago 12m 11.5M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 d42a6ffd2e66 2024-06-21T01:55:57.999 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (11m) 18s ago 11m 58.3M 2992M 17.2.0 e1d6a67b021e 9b76c4cf6035 2024-06-21T01:55:57.999 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (11m) 18s ago 11m 53.5M 2992M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:55:57.999 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (10m) 18s ago 10m 58.2M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:55:58.000 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (4m) 18s ago 10m 75.0M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:55:58.000 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (8m) 50s ago 8m 76.8M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:55:58.000 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (8m) 18s ago 8m 73.3M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:55:58.000 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (8m) 50s ago 8m 74.5M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:55:58.000 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (8m) 18s ago 8m 74.5M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:55:58.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:57 smithi173 ceph-mon[57540]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:58.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:57 smithi092 ceph-mon[64276]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:58.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:57 smithi092 ceph-mon[66756]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:55:58.389 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:55:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:55:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:55:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:55:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 5 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 10 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:55:58.390 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:55:58.718 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:55:58.718 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:55:58.718 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:55:58.718 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:55:58.718 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:55:58.718 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr", 2024-06-21T01:55:58.718 INFO:teuthology.orchestra.run.smithi092.stdout: "mon" 2024-06-21T01:55:58.718 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:55:58.718 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "10/23 daemons upgraded", 2024-06-21T01:55:58.719 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading osd daemons", 2024-06-21T01:55:58.719 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:55:58.719 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:55:58.937 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:58.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-06-21T01:55:58.937 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 01:55:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T01:55:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)\", cluster=\"22eef780-2f6f-11ef-bc9d-c7b262605968\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"smithi092:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.92\", device_class=\"ssd\", hostname=\"smithi092\", instance=\"172.21.15.173:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.92\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-21T01:55:59.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:58 smithi092 ceph-mon[64276]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:59.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:58 smithi092 ceph-mon[64276]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:59.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:58 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/4276855067' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:59.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:58 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:55:59.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:58 smithi092 ceph-mon[66756]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:59.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:58 smithi092 ceph-mon[66756]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:59.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:58 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/4276855067' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:59.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:58 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:55:59.119 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:55:59.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:58 smithi173 ceph-mon[57540]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:59.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:58 smithi173 ceph-mon[57540]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:55:59.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:58 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/4276855067' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:55:59.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:58 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:55:59.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:55:59] "GET /metrics HTTP/1.1" 200 38436 "" "Prometheus/2.33.4" 2024-06-21T01:56:00.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:59 smithi173 ceph-mon[57540]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:00.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:59 smithi173 ceph-mon[57540]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:00.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:59 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:56:00.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:59 smithi173 ceph-mon[57540]: Upgrade: osd.5 is safe to restart 2024-06-21T01:56:00.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:59 smithi173 ceph-mon[57540]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:56:00.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:55:59 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/3502339379' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:56:00.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[64276]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:00.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[64276]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:00.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:56:00.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[64276]: Upgrade: osd.5 is safe to restart 2024-06-21T01:56:00.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[64276]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:56:00.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/3502339379' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:56:00.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[66756]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:00.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[66756]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:00.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-21T01:56:00.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[66756]: Upgrade: osd.5 is safe to restart 2024-06-21T01:56:00.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[66756]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:56:00.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:55:59 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/3502339379' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:56:01.540 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:01 smithi173 ceph-mon[57540]: Upgrade: Updating osd.5 2024-06-21T01:56:01.540 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:01 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:01.541 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:01 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-21T01:56:01.541 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:01 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:01.541 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:01 smithi173 ceph-mon[57540]: Deploying daemon osd.5 on smithi173 2024-06-21T01:56:01.541 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:01 smithi173 ceph-mon[57540]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:56:01.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[64276]: Upgrade: Updating osd.5 2024-06-21T01:56:01.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:01.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-21T01:56:01.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:01.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[64276]: Deploying daemon osd.5 on smithi173 2024-06-21T01:56:01.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[64276]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:56:01.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[66756]: Upgrade: Updating osd.5 2024-06-21T01:56:01.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:01.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-21T01:56:01.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:01.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[66756]: Deploying daemon osd.5 on smithi173 2024-06-21T01:56:01.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:01 smithi092 ceph-mon[66756]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:56:03.156 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:03 smithi173 ceph-mon[57540]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:56:03.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:03 smithi092 ceph-mon[64276]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:56:03.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:03 smithi092 ceph-mon[66756]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:56:03.429 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:03 smithi173 systemd[1]: Stopping Ceph osd.5 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:56:03.728 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[32238]: 2024-06-21T01:56:03.431+0000 7fc90bedc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:56:03.728 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[32238]: 2024-06-21T01:56:03.432+0000 7fc90bedc700 -1 osd.5 131 *** Got signal Terminated *** 2024-06-21T01:56:03.728 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[32238]: 2024-06-21T01:56:03.432+0000 7fc90bedc700 -1 osd.5 131 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T01:56:04.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:04 smithi092 ceph-mon[64276]: osd.5 marked itself down and dead 2024-06-21T01:56:04.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:04 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:04.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:04 smithi092 ceph-mon[66756]: osd.5 marked itself down and dead 2024-06-21T01:56:04.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:04 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:04.389 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:04 smithi173 ceph-mon[57540]: osd.5 marked itself down and dead 2024-06-21T01:56:04.389 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:04 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:04.662 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:04 smithi173 podman[62607]: 2024-06-21 01:56:04.39257888 +0000 UTC m=+1.087024760 container died 9b76c4cf6035bb67589f55c7f0bb9a60a963da870d7f7a3861463327ca7e816e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5, distribution-scope=public, ceph=True, io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, GIT_CLEAN=True, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, architecture=x86_64, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870) 2024-06-21T01:56:04.978 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:04 smithi173 podman[62607]: 2024-06-21 01:56:04.756580835 +0000 UTC m=+1.451026715 container cleanup 9b76c4cf6035bb67589f55c7f0bb9a60a963da870d7f7a3861463327ca7e816e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_BRANCH=HEAD, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8) 2024-06-21T01:56:04.978 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:04 smithi173 bash[62607]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5 2024-06-21T01:56:05.346 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:05 smithi173 ceph-mon[57540]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:56:05.347 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:05 smithi173 ceph-mon[57540]: osdmap e132: 8 total, 7 up, 8 in 2024-06-21T01:56:05.347 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:05 smithi173 ceph-mon[57540]: pgmap v146: 161 pgs: 5 peering, 21 stale+active+clean, 135 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:56:05.347 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:05 smithi173 podman[62625]: 2024-06-21 01:56:05.041275839 +0000 UTC m=+0.647693022 container remove 9b76c4cf6035bb67589f55c7f0bb9a60a963da870d7f7a3861463327ca7e816e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, ceph=True, name=centos-stream, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , version=8, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870) 2024-06-21T01:56:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:05 smithi092 ceph-mon[64276]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:56:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:05 smithi092 ceph-mon[64276]: osdmap e132: 8 total, 7 up, 8 in 2024-06-21T01:56:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:05 smithi092 ceph-mon[64276]: pgmap v146: 161 pgs: 5 peering, 21 stale+active+clean, 135 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:56:05.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:05 smithi092 ceph-mon[66756]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:56:05.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:05 smithi092 ceph-mon[66756]: osdmap e132: 8 total, 7 up, 8 in 2024-06-21T01:56:05.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:05 smithi092 ceph-mon[66756]: pgmap v146: 161 pgs: 5 peering, 21 stale+active+clean, 135 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:56:05.674 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:05 smithi173 podman[62711]: 2024-06-21 01:56:05.350554825 +0000 UTC m=+0.105079748 container create ab8cdee9f5eb73b88085a811ace06f80060764e23f0ab9006d25ce3b14884b18 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux ) 2024-06-21T01:56:05.674 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:05 smithi173 podman[62711]: 2024-06-21 01:56:05.263522852 +0000 UTC m=+0.018047791 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:56:05.674 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:05 smithi173 podman[62711]: 2024-06-21 01:56:05.578195655 +0000 UTC m=+0.332720587 container init ab8cdee9f5eb73b88085a811ace06f80060764e23f0ab9006d25ce3b14884b18 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-deactivate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-21T01:56:05.674 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:05 smithi173 podman[62711]: 2024-06-21 01:56:05.582633251 +0000 UTC m=+0.337158185 container start ab8cdee9f5eb73b88085a811ace06f80060764e23f0ab9006d25ce3b14884b18 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:56:05.674 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:05 smithi173 podman[62711]: 2024-06-21 01:56:05.678203044 +0000 UTC m=+0.432727974 container attach ab8cdee9f5eb73b88085a811ace06f80060764e23f0ab9006d25ce3b14884b18 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-06-21T01:56:05.978 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:05 smithi173 podman[62711]: 2024-06-21 01:56:05.846023308 +0000 UTC m=+0.600548240 container died ab8cdee9f5eb73b88085a811ace06f80060764e23f0ab9006d25ce3b14884b18 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-21T01:56:06.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:06 smithi092 ceph-mon[64276]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-21T01:56:06.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:06 smithi092 ceph-mon[64276]: osdmap e133: 8 total, 7 up, 8 in 2024-06-21T01:56:06.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:06 smithi092 ceph-mon[66756]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-21T01:56:06.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:06 smithi092 ceph-mon[66756]: osdmap e133: 8 total, 7 up, 8 in 2024-06-21T01:56:06.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:06 smithi173 ceph-mon[57540]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-21T01:56:06.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:06 smithi173 ceph-mon[57540]: osdmap e133: 8 total, 7 up, 8 in 2024-06-21T01:56:06.938 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:06 smithi173 podman[62735]: 2024-06-21 01:56:06.507260485 +0000 UTC m=+0.655763814 container remove ab8cdee9f5eb73b88085a811ace06f80060764e23f0ab9006d25ce3b14884b18 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-deactivate, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True) 2024-06-21T01:56:06.938 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:06 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.5.service: Deactivated successfully. 2024-06-21T01:56:06.939 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:06 smithi173 systemd[1]: Stopped Ceph osd.5 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:56:06.939 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:06 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.5.service: Consumed 8.285s CPU time. 2024-06-21T01:56:06.939 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:06 smithi173 systemd[1]: Starting Ceph osd.5 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:56:07.198 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:07 smithi173 ceph-mon[57540]: pgmap v148: 161 pgs: 20 active+undersized, 15 peering, 6 stale+active+clean, 9 active+undersized+degraded, 111 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 32/723 objects degraded (4.426%) 2024-06-21T01:56:07.199 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:06 smithi173 podman[62835]: 2024-06-21 01:56:06.941889355 +0000 UTC m=+0.113917364 container create 2707441d783191f6f3c1958968128d6a2a6ea3adc5aea7d0e45af3a1c12432c8 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-06-21T01:56:07.199 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:06 smithi173 podman[62835]: 2024-06-21 01:56:06.844963204 +0000 UTC m=+0.016991213 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:56:07.199 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 podman[62835]: 2024-06-21 01:56:07.202574378 +0000 UTC m=+0.374602386 container init 2707441d783191f6f3c1958968128d6a2a6ea3adc5aea7d0e45af3a1c12432c8 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-06-21T01:56:07.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:07 smithi092 ceph-mon[64276]: pgmap v148: 161 pgs: 20 active+undersized, 15 peering, 6 stale+active+clean, 9 active+undersized+degraded, 111 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 32/723 objects degraded (4.426%) 2024-06-21T01:56:07.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:07 smithi092 ceph-mon[66756]: pgmap v148: 161 pgs: 20 active+undersized, 15 peering, 6 stale+active+clean, 9 active+undersized+degraded, 111 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 32/723 objects degraded (4.426%) 2024-06-21T01:56:07.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 podman[62835]: 2024-06-21 01:56:07.206788313 +0000 UTC m=+0.378816329 container start 2707441d783191f6f3c1958968128d6a2a6ea3adc5aea7d0e45af3a1c12432c8 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:56:07.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 podman[62835]: 2024-06-21 01:56:07.269268446 +0000 UTC m=+0.441296471 container attach 2707441d783191f6f3c1958968128d6a2a6ea3adc5aea7d0e45af3a1c12432c8 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-21T01:56:07.902 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate[62850]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 bash[62835]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate[62850]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 bash[62835]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate[62850]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 bash[62835]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate[62850]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 bash[62835]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate[62850]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 bash[62835]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate[62850]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 bash[62835]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-21T01:56:07.903 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate[62850]: --> ceph-volume raw activate successful for osd ID: 5 2024-06-21T01:56:07.904 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 bash[62835]: --> ceph-volume raw activate successful for osd ID: 5 2024-06-21T01:56:07.904 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:07 smithi173 podman[62835]: 2024-06-21 01:56:07.62325722 +0000 UTC m=+0.795285228 container died 2707441d783191f6f3c1958968128d6a2a6ea3adc5aea7d0e45af3a1c12432c8 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0) 2024-06-21T01:56:08.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:08 smithi173 ceph-mon[57540]: Health check failed: Degraded data redundancy: 32/723 objects degraded (4.426%), 9 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:08.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:08 smithi092 ceph-mon[64276]: Health check failed: Degraded data redundancy: 32/723 objects degraded (4.426%), 9 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:08.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:08 smithi092 ceph-mon[66756]: Health check failed: Degraded data redundancy: 32/723 objects degraded (4.426%), 9 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:08.575 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:08 smithi173 podman[62979]: 2024-06-21 01:56:08.273646059 +0000 UTC m=+0.644761058 container remove 2707441d783191f6f3c1958968128d6a2a6ea3adc5aea7d0e45af3a1c12432c8 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-21T01:56:08.575 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:08 smithi173 podman[63038]: 2024-06-21 01:56:08.482834927 +0000 UTC m=+0.111481266 container create b6050a92252eb67ab6c60a0ba73c332b601201e9ffda0e788116a4c24e2dc8e7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:56:08.576 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:08 smithi173 podman[63038]: 2024-06-21 01:56:08.388382421 +0000 UTC m=+0.017028768 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:56:08.978 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:08 smithi173 podman[63038]: 2024-06-21 01:56:08.676643268 +0000 UTC m=+0.305289623 container init b6050a92252eb67ab6c60a0ba73c332b601201e9ffda0e788116a4c24e2dc8e7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux ) 2024-06-21T01:56:08.978 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:08 smithi173 podman[63038]: 2024-06-21 01:56:08.679794984 +0000 UTC m=+0.308441325 container start b6050a92252eb67ab6c60a0ba73c332b601201e9ffda0e788116a4c24e2dc8e7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-21T01:56:08.978 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:08 smithi173 ceph-osd[63057]: -- 172.21.15.173:0/3691811414 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f02993a40 con 0x556f02971c00 2024-06-21T01:56:08.978 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:08 smithi173 bash[63038]: b6050a92252eb67ab6c60a0ba73c332b601201e9ffda0e788116a4c24e2dc8e7 2024-06-21T01:56:08.979 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:08 smithi173 systemd[1]: Started Ceph osd.5 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:56:09.479 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T01:56:09.218+0000 7fc96e438640 -1 Falling back to public interface 2024-06-21T01:56:09.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:56:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:56:09] "GET /metrics HTTP/1.1" 200 38436 "" "Prometheus/2.33.4" 2024-06-21T01:56:10.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:09 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:10.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:09 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:10.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:09 smithi092 ceph-mon[64276]: pgmap v149: 161 pgs: 27 active+undersized, 15 peering, 18 active+undersized+degraded, 101 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-06-21T01:56:10.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:09 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:10.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:09 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:10.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:09 smithi092 ceph-mon[66756]: pgmap v149: 161 pgs: 27 active+undersized, 15 peering, 18 active+undersized+degraded, 101 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-06-21T01:56:10.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:09 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:10.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:09 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:10.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:09 smithi173 ceph-mon[57540]: pgmap v149: 161 pgs: 27 active+undersized, 15 peering, 18 active+undersized+degraded, 101 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-06-21T01:56:12.228 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T01:56:11.915+0000 7fc96e438640 -1 osd.5 0 read_superblock omap replica is missing. 2024-06-21T01:56:12.228 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T01:56:11.939+0000 7fc96e438640 -1 osd.5 131 log_to_monitors true 2024-06-21T01:56:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:11 smithi173 ceph-mon[57540]: pgmap v150: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-21T01:56:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:12.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:11 smithi173 ceph-mon[57540]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-06-21T01:56:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:11 smithi092 ceph-mon[64276]: pgmap v150: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-21T01:56:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:11 smithi092 ceph-mon[64276]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-06-21T01:56:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:11 smithi092 ceph-mon[66756]: pgmap v150: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-21T01:56:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:11 smithi092 ceph-mon[66756]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-06-21T01:56:13.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:12 smithi173 ceph-mon[57540]: from='osd.5 [v2:172.21.15.173:6808/4013831737,v1:172.21.15.173:6809/4013831737]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:56:13.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:12 smithi173 ceph-mon[57540]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:56:13.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:12 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:13.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:12 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:12 smithi092 ceph-mon[64276]: from='osd.5 [v2:172.21.15.173:6808/4013831737,v1:172.21.15.173:6809/4013831737]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:56:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:12 smithi092 ceph-mon[64276]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:56:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:12 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:12 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:13.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:12 smithi092 ceph-mon[66756]: from='osd.5 [v2:172.21.15.173:6808/4013831737,v1:172.21.15.173:6809/4013831737]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:56:13.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:12 smithi092 ceph-mon[66756]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-21T01:56:13.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:12 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:13.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:12 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:13.728 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 01:56:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T01:56:13.384+0000 7fc9621db640 -1 osd.5 131 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:56:14.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:13 smithi173 ceph-mon[57540]: pgmap v151: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-21T01:56:14.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:13 smithi173 ceph-mon[57540]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-21T01:56:14.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:13 smithi173 ceph-mon[57540]: osdmap e134: 8 total, 7 up, 8 in 2024-06-21T01:56:14.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:13 smithi173 ceph-mon[57540]: from='osd.5 [v2:172.21.15.173:6808/4013831737,v1:172.21.15.173:6809/4013831737]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:14.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:13 smithi173 ceph-mon[57540]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:14.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:13 smithi173 ceph-mon[57540]: from='osd.5 ' entity='osd.5' 2024-06-21T01:56:14.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:14.136 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:14.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[64276]: pgmap v151: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-21T01:56:14.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[64276]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-21T01:56:14.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[64276]: osdmap e134: 8 total, 7 up, 8 in 2024-06-21T01:56:14.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[64276]: from='osd.5 [v2:172.21.15.173:6808/4013831737,v1:172.21.15.173:6809/4013831737]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:14.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[64276]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:14.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[64276]: from='osd.5 ' entity='osd.5' 2024-06-21T01:56:14.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:14.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:14.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[66756]: pgmap v151: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-21T01:56:14.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[66756]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-21T01:56:14.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[66756]: osdmap e134: 8 total, 7 up, 8 in 2024-06-21T01:56:14.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[66756]: from='osd.5 [v2:172.21.15.173:6808/4013831737,v1:172.21.15.173:6809/4013831737]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:14.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[66756]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:14.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[66756]: from='osd.5 ' entity='osd.5' 2024-06-21T01:56:14.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:14.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:15.160 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:14 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:15.160 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:14 smithi173 ceph-mon[57540]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:56:15.160 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:14 smithi173 ceph-mon[57540]: osd.5 [v2:172.21.15.173:6808/4013831737,v1:172.21.15.173:6809/4013831737] boot 2024-06-21T01:56:15.160 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:14 smithi173 ceph-mon[57540]: osdmap e135: 8 total, 8 up, 8 in 2024-06-21T01:56:15.160 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:14 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:56:15.160 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:14 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 81/723 objects degraded (11.203%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:15.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:15.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[64276]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:56:15.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[64276]: osd.5 [v2:172.21.15.173:6808/4013831737,v1:172.21.15.173:6809/4013831737] boot 2024-06-21T01:56:15.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[64276]: osdmap e135: 8 total, 8 up, 8 in 2024-06-21T01:56:15.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:56:15.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 81/723 objects degraded (11.203%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:15.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:15.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[66756]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:56:15.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[66756]: osd.5 [v2:172.21.15.173:6808/4013831737,v1:172.21.15.173:6809/4013831737] boot 2024-06-21T01:56:15.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[66756]: osdmap e135: 8 total, 8 up, 8 in 2024-06-21T01:56:15.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-21T01:56:15.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:14 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 81/723 objects degraded (11.203%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:15 smithi173 ceph-mon[57540]: pgmap v154: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-21T01:56:16.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:15 smithi173 ceph-mon[57540]: osdmap e136: 8 total, 8 up, 8 in 2024-06-21T01:56:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:15 smithi092 ceph-mon[64276]: pgmap v154: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-21T01:56:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:15 smithi092 ceph-mon[64276]: osdmap e136: 8 total, 8 up, 8 in 2024-06-21T01:56:16.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:15 smithi092 ceph-mon[66756]: pgmap v154: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-21T01:56:16.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:15 smithi092 ceph-mon[66756]: osdmap e136: 8 total, 8 up, 8 in 2024-06-21T01:56:17.101 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:56:16 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:56:16.745+0000 7f9965fe7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-21T01:56:17.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:17.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:17.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:17.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:56:17.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:56:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-21T01:56:17.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:17 smithi173 ceph-mon[57540]: pgmap v156: 161 pgs: 6 peering, 22 active+undersized, 14 active+undersized+degraded, 119 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/723 objects degraded (7.607%) 2024-06-21T01:56:18.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:18.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:18.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:18.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:56:18.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:18.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:56:18.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:18.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:18.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:18.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:18.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:18.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-21T01:56:18.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[64276]: pgmap v156: 161 pgs: 6 peering, 22 active+undersized, 14 active+undersized+degraded, 119 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/723 objects degraded (7.607%) 2024-06-21T01:56:18.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:18.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:18.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:18.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:56:18.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:18.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:56:18.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:18.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:18.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:18.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:18.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:18.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-21T01:56:18.107 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:17 smithi092 ceph-mon[66756]: pgmap v156: 161 pgs: 6 peering, 22 active+undersized, 14 active+undersized+degraded, 119 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/723 objects degraded (7.607%) 2024-06-21T01:56:19.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:19 smithi092 ceph-mon[64276]: pgmap v157: 161 pgs: 6 peering, 4 active+undersized, 151 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-06-21T01:56:19.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:19 smithi092 ceph-mon[64276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 55/723 objects degraded (7.607%), 14 pgs degraded) 2024-06-21T01:56:19.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:19 smithi092 ceph-mon[64276]: Cluster is now healthy 2024-06-21T01:56:19.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:19 smithi092 ceph-mon[66756]: pgmap v157: 161 pgs: 6 peering, 4 active+undersized, 151 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-06-21T01:56:19.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:19 smithi092 ceph-mon[66756]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 55/723 objects degraded (7.607%), 14 pgs degraded) 2024-06-21T01:56:19.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:19 smithi092 ceph-mon[66756]: Cluster is now healthy 2024-06-21T01:56:19.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:19 smithi173 ceph-mon[57540]: pgmap v157: 161 pgs: 6 peering, 4 active+undersized, 151 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-06-21T01:56:19.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:19 smithi173 ceph-mon[57540]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 55/723 objects degraded (7.607%), 14 pgs degraded) 2024-06-21T01:56:19.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:19 smithi173 ceph-mon[57540]: Cluster is now healthy 2024-06-21T01:56:19.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:56:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:56:19] "GET /metrics HTTP/1.1" 200 38517 "" "Prometheus/2.33.4" 2024-06-21T01:56:21.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:21 smithi173 ceph-mon[57540]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:56:21.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:21 smithi092 ceph-mon[64276]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:56:21.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:21 smithi092 ceph-mon[66756]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:56:23.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:23 smithi092 ceph-mon[64276]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-06-21T01:56:23.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:23 smithi092 ceph-mon[66756]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-06-21T01:56:23.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:23 smithi173 ceph-mon[57540]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-06-21T01:56:24.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:24 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:24.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:24 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:24.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:24 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:25.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:25 smithi173 ceph-mon[57540]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-06-21T01:56:25.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:25 smithi092 ceph-mon[64276]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-06-21T01:56:25.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:25 smithi092 ceph-mon[66756]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-06-21T01:56:27.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:27.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:27.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:27.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:27.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:27.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:27.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:27.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:27.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:27 smithi173 ceph-mon[57540]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:56:28.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:27 smithi092 ceph-mon[64276]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:56:28.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:27 smithi092 ceph-mon[66756]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-06-21T01:56:29.474 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:56:29.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:56:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:56:29] "GET /metrics HTTP/1.1" 200 38520 "" "Prometheus/2.33.4" 2024-06-21T01:56:30.123 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (5m) 82s ago 9m 21.0M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (5m) 17s ago 9m 55.1M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (4m) 82s ago 9m 52.6M - 3.5 e1d6a67b021e 3192bc59456b 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (4m) 17s ago 14m 440M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (5m) 82s ago 15m 502M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (4m) 82s ago 16m 49.7M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (3m) 17s ago 14m 39.3M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (4m) 82s ago 14m 42.0M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (5m) 82s ago 10m 16.8M - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (5m) 17s ago 10m 18.1M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (3m) 82s ago 14m 52.6M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 7c252b0f2dfd 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (2m) 82s ago 13m 44.8M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 9ad8cda61390 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (119s) 82s ago 13m 49.8M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 0fe024e7dad9 2024-06-21T01:56:30.124 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (87s) 82s ago 12m 14.5M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 f0817e4dfec7 2024-06-21T01:56:30.125 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (54s) 17s ago 12m 46.2M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 d42a6ffd2e66 2024-06-21T01:56:30.125 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (21s) 17s ago 12m 12.1M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 b6050a92252e 2024-06-21T01:56:30.125 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (11m) 17s ago 11m 55.5M 2992M 17.2.0 e1d6a67b021e a999e7cea902 2024-06-21T01:56:30.125 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (11m) 17s ago 11m 59.1M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:56:30.125 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (4m) 17s ago 10m 75.0M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:56:30.125 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (9m) 82s ago 9m 76.8M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:56:30.125 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (9m) 17s ago 9m 73.4M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:56:30.125 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (9m) 82s ago 9m 74.5M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:56:30.125 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (9m) 17s ago 9m 74.7M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:56:30.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:29 smithi173 ceph-mon[57540]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-06-21T01:56:30.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:29 smithi092 ceph-mon[64276]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-06-21T01:56:30.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:29 smithi092 ceph-mon[66756]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-06-21T01:56:30.514 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:56:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:56:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:56:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:56:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:56:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 6 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 11 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:56:30.515 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:56:30.846 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:56:30.846 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:56:30.846 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:56:30.846 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:56:30.846 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:56:30.846 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr", 2024-06-21T01:56:30.846 INFO:teuthology.orchestra.run.smithi092.stdout: "mon" 2024-06-21T01:56:30.846 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:56:30.846 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "11/23 daemons upgraded", 2024-06-21T01:56:30.846 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading osd daemons", 2024-06-21T01:56:30.847 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:56:30.847 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:56:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:30 smithi092 ceph-mon[64276]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:30 smithi092 ceph-mon[64276]: from='client.44392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:30 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/1180090346' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:31.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:30 smithi092 ceph-mon[66756]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:31.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:30 smithi092 ceph-mon[66756]: from='client.44392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:31.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:30 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/1180090346' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:31.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:30 smithi173 ceph-mon[57540]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:31.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:30 smithi173 ceph-mon[57540]: from='client.44392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:31.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:30 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/1180090346' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:31.237 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:56:32.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:31 smithi173 ceph-mon[57540]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:32.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:31 smithi173 ceph-mon[57540]: from='client.44410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:31 smithi173 ceph-mon[57540]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:56:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:31 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/3144745244' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:56:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:31 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:32.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:31 smithi092 ceph-mon[64276]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:32.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:31 smithi092 ceph-mon[64276]: from='client.44410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:32.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:31 smithi092 ceph-mon[64276]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:56:32.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:31 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/3144745244' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:56:32.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:31 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:32.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:31 smithi092 ceph-mon[66756]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:32.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:31 smithi092 ceph-mon[66756]: from='client.44410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:56:32.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:31 smithi092 ceph-mon[66756]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:56:32.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:31 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/3144745244' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:56:32.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:31 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:33.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:32 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:33.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:32 smithi173 ceph-mon[57540]: Upgrade: osd.6 is safe to restart 2024-06-21T01:56:33.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:32 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:33.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:32 smithi092 ceph-mon[66756]: Upgrade: osd.6 is safe to restart 2024-06-21T01:56:33.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:32 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-21T01:56:33.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:32 smithi092 ceph-mon[64276]: Upgrade: osd.6 is safe to restart 2024-06-21T01:56:34.141 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:33 smithi173 ceph-mon[57540]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:56:34.142 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:34.142 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-21T01:56:34.142 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:33 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:34.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:33 smithi092 ceph-mon[66756]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:56:34.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:34.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-21T01:56:34.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:33 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:34.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:33 smithi092 ceph-mon[64276]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:56:34.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:34.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-21T01:56:34.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:33 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:35.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:34 smithi173 ceph-mon[57540]: Upgrade: Updating osd.6 2024-06-21T01:56:35.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:34 smithi173 ceph-mon[57540]: Deploying daemon osd.6 on smithi173 2024-06-21T01:56:35.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:34 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:35.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:34 smithi173 ceph-mon[57540]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:56:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:34 smithi092 ceph-mon[66756]: Upgrade: Updating osd.6 2024-06-21T01:56:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:34 smithi092 ceph-mon[66756]: Deploying daemon osd.6 on smithi173 2024-06-21T01:56:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:34 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:35.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:34 smithi092 ceph-mon[66756]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:56:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:34 smithi092 ceph-mon[64276]: Upgrade: Updating osd.6 2024-06-21T01:56:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:34 smithi092 ceph-mon[64276]: Deploying daemon osd.6 on smithi173 2024-06-21T01:56:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:34 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:34 smithi092 ceph-mon[64276]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:56:36.255 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:36 smithi173 systemd[1]: Stopping Ceph osd.6 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:56:36.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:36 smithi092 ceph-mon[64276]: osd.6 marked itself down and dead 2024-06-21T01:56:36.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:36 smithi092 ceph-mon[66756]: osd.6 marked itself down and dead 2024-06-21T01:56:36.643 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:36 smithi173 ceph-mon[57540]: osd.6 marked itself down and dead 2024-06-21T01:56:36.643 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[35118]: 2024-06-21T01:56:36.257+0000 7f7b65a58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:56:36.643 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[35118]: 2024-06-21T01:56:36.257+0000 7f7b65a58700 -1 osd.6 136 *** Got signal Terminated *** 2024-06-21T01:56:36.643 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:36 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[35118]: 2024-06-21T01:56:36.257+0000 7f7b65a58700 -1 osd.6 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T01:56:36.921 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:36 smithi173 podman[64898]: 2024-06-21 01:56:36.646610997 +0000 UTC m=+0.483197090 container died a999e7cea9029cf314029402e82fea854e62d91dc3a2bdde56b9f6ec5bfaea0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, release=754, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, distribution-scope=public, version=8, io.buildah.version=1.19.8, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.openshift.expose-services=, architecture=x86_64, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-06-21T01:56:37.192 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:36 smithi173 podman[64898]: 2024-06-21 01:56:36.99937705 +0000 UTC m=+0.835963143 container cleanup a999e7cea9029cf314029402e82fea854e62d91dc3a2bdde56b9f6ec5bfaea0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6, vendor=Red Hat, Inc., architecture=x86_64, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, release=754, ceph=True, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, name=centos-stream, version=8, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD) 2024-06-21T01:56:37.192 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:36 smithi173 bash[64898]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6 2024-06-21T01:56:37.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:37 smithi173 ceph-mon[57540]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:56:37.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:37 smithi173 ceph-mon[57540]: osdmap e137: 8 total, 7 up, 8 in 2024-06-21T01:56:37.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:37 smithi173 ceph-mon[57540]: pgmap v167: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:56:37.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:37 smithi173 ceph-mon[57540]: osdmap e138: 8 total, 7 up, 8 in 2024-06-21T01:56:37.479 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:37 smithi173 podman[64923]: 2024-06-21 01:56:37.301132635 +0000 UTC m=+0.650128566 container remove a999e7cea9029cf314029402e82fea854e62d91dc3a2bdde56b9f6ec5bfaea0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, architecture=x86_64, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.buildah.version=1.19.8) 2024-06-21T01:56:37.604 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:37 smithi092 ceph-mon[66756]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:56:37.604 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:37 smithi092 ceph-mon[66756]: osdmap e137: 8 total, 7 up, 8 in 2024-06-21T01:56:37.604 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:37 smithi092 ceph-mon[66756]: pgmap v167: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:56:37.604 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:37 smithi092 ceph-mon[66756]: osdmap e138: 8 total, 7 up, 8 in 2024-06-21T01:56:37.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:37 smithi092 ceph-mon[64276]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:56:37.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:37 smithi092 ceph-mon[64276]: osdmap e137: 8 total, 7 up, 8 in 2024-06-21T01:56:37.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:37 smithi092 ceph-mon[64276]: pgmap v167: 161 pgs: 10 peering, 12 stale+active+clean, 139 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:56:37.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:37 smithi092 ceph-mon[64276]: osdmap e138: 8 total, 7 up, 8 in 2024-06-21T01:56:37.943 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:37 smithi173 podman[65013]: 2024-06-21 01:56:37.547352814 +0000 UTC m=+0.017157078 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:56:37.944 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:37 smithi173 podman[65013]: 2024-06-21 01:56:37.660487318 +0000 UTC m=+0.130291576 container create 004e5c4a127be20f54c9fcaf00f6e416cab596bb266368f0e6c920858f814b9c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, ceph=True, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-21T01:56:37.944 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:37 smithi173 podman[65013]: 2024-06-21 01:56:37.8710367 +0000 UTC m=+0.340840958 container init 004e5c4a127be20f54c9fcaf00f6e416cab596bb266368f0e6c920858f814b9c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:56:37.944 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:37 smithi173 podman[65013]: 2024-06-21 01:56:37.874304047 +0000 UTC m=+0.344108305 container start 004e5c4a127be20f54c9fcaf00f6e416cab596bb266368f0e6c920858f814b9c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, ceph=True, CEPH_POINT_RELEASE=) 2024-06-21T01:56:37.944 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:37 smithi173 podman[65013]: 2024-06-21 01:56:37.946053593 +0000 UTC m=+0.415857866 container attach 004e5c4a127be20f54c9fcaf00f6e416cab596bb266368f0e6c920858f814b9c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, GIT_CLEAN=True, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-06-21T01:56:38.228 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:38 smithi173 podman[65013]: 2024-06-21 01:56:38.14125417 +0000 UTC m=+0.611058431 container died 004e5c4a127be20f54c9fcaf00f6e416cab596bb266368f0e6c920858f814b9c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-21T01:56:38.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:38 smithi092 ceph-mon[64276]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-21T01:56:38.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:38 smithi092 ceph-mon[66756]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-21T01:56:38.653 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:38 smithi173 ceph-mon[57540]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-21T01:56:38.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:38 smithi173 podman[65035]: 2024-06-21 01:56:38.657819228 +0000 UTC m=+0.507578940 container remove 004e5c4a127be20f54c9fcaf00f6e416cab596bb266368f0e6c920858f814b9c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, ceph=True) 2024-06-21T01:56:38.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:38 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.6.service: Deactivated successfully. 2024-06-21T01:56:38.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:38 smithi173 systemd[1]: Stopped Ceph osd.6 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:56:38.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:38 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.6.service: Consumed 7.809s CPU time. 2024-06-21T01:56:38.979 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:38 smithi173 systemd[1]: Starting Ceph osd.6 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:56:39.366 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:39 smithi173 ceph-mon[57540]: pgmap v169: 161 pgs: 7 active+undersized, 21 peering, 7 stale+active+clean, 6 active+undersized+degraded, 120 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 16/723 objects degraded (2.213%) 2024-06-21T01:56:39.367 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 podman[65133]: 2024-06-21 01:56:38.999236579 +0000 UTC m=+0.017945353 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:56:39.367 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 podman[65133]: 2024-06-21 01:56:39.109563079 +0000 UTC m=+0.128271888 container create b00e14011b3119df93a0e8c4c2cdebf18bbca90cb52c2bbfc3448c97184eb2bb (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618) 2024-06-21T01:56:39.367 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 podman[65133]: 2024-06-21 01:56:39.37079018 +0000 UTC m=+0.389498964 container init b00e14011b3119df93a0e8c4c2cdebf18bbca90cb52c2bbfc3448c97184eb2bb (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-21T01:56:39.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:39 smithi092 ceph-mon[64276]: pgmap v169: 161 pgs: 7 active+undersized, 21 peering, 7 stale+active+clean, 6 active+undersized+degraded, 120 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 16/723 objects degraded (2.213%) 2024-06-21T01:56:39.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:39 smithi092 ceph-mon[66756]: pgmap v169: 161 pgs: 7 active+undersized, 21 peering, 7 stale+active+clean, 6 active+undersized+degraded, 120 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 16/723 objects degraded (2.213%) 2024-06-21T01:56:39.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:56:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:56:39] "GET /metrics HTTP/1.1" 200 38520 "" "Prometheus/2.33.4" 2024-06-21T01:56:39.728 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 podman[65133]: 2024-06-21 01:56:39.374483603 +0000 UTC m=+0.393192369 container start b00e14011b3119df93a0e8c4c2cdebf18bbca90cb52c2bbfc3448c97184eb2bb (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=) 2024-06-21T01:56:39.728 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 podman[65133]: 2024-06-21 01:56:39.470391199 +0000 UTC m=+0.489099973 container attach b00e14011b3119df93a0e8c4c2cdebf18bbca90cb52c2bbfc3448c97184eb2bb (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate, org.label-schema.build-date=20240618, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-21T01:56:40.096 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate[65147]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-21T01:56:40.096 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 bash[65133]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-21T01:56:40.096 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate[65147]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-21T01:56:40.096 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 bash[65133]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-21T01:56:40.096 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate[65147]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-21T01:56:40.096 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 bash[65133]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-21T01:56:40.096 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate[65147]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-21T01:56:40.096 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 bash[65133]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-21T01:56:40.096 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate[65147]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-06-21T01:56:40.096 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 bash[65133]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-06-21T01:56:40.096 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate[65147]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-21T01:56:40.097 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 bash[65133]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-21T01:56:40.097 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate[65147]: --> ceph-volume raw activate successful for osd ID: 6 2024-06-21T01:56:40.097 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 bash[65133]: --> ceph-volume raw activate successful for osd ID: 6 2024-06-21T01:56:40.097 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:39 smithi173 podman[65133]: 2024-06-21 01:56:39.795004706 +0000 UTC m=+0.813713477 container died b00e14011b3119df93a0e8c4c2cdebf18bbca90cb52c2bbfc3448c97184eb2bb (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-06-21T01:56:40.411 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:40 smithi173 ceph-mon[57540]: Health check failed: Degraded data redundancy: 16/723 objects degraded (2.213%), 6 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:40.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:40 smithi092 ceph-mon[64276]: Health check failed: Degraded data redundancy: 16/723 objects degraded (2.213%), 6 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:40.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:40 smithi092 ceph-mon[66756]: Health check failed: Degraded data redundancy: 16/723 objects degraded (2.213%), 6 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:40.728 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:40 smithi173 podman[65267]: 2024-06-21 01:56:40.415689404 +0000 UTC m=+0.615253239 container remove b00e14011b3119df93a0e8c4c2cdebf18bbca90cb52c2bbfc3448c97184eb2bb (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-activate, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-21T01:56:40.728 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:40 smithi173 podman[65327]: 2024-06-21 01:56:40.624997559 +0000 UTC m=+0.111810484 container create dffac360ac20aabb4ccf2443b0f04daa47db3a0184c9855e1471b55dfb0fd69c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81) 2024-06-21T01:56:40.728 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:40 smithi173 podman[65327]: 2024-06-21 01:56:40.530241728 +0000 UTC m=+0.017054610 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:56:41.228 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:40 smithi173 podman[65327]: 2024-06-21 01:56:40.844016207 +0000 UTC m=+0.330829086 container init dffac360ac20aabb4ccf2443b0f04daa47db3a0184c9855e1471b55dfb0fd69c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6, RELEASE=squid-release-c65be81, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-21T01:56:41.228 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:40 smithi173 podman[65327]: 2024-06-21 01:56:40.847180106 +0000 UTC m=+0.333992994 container start dffac360ac20aabb4ccf2443b0f04daa47db3a0184c9855e1471b55dfb0fd69c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-21T01:56:41.228 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:40 smithi173 ceph-osd[65346]: -- 172.21.15.173:0/1594543424 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5568b7b47a40 con 0x5568b8906000 2024-06-21T01:56:41.228 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:40 smithi173 bash[65327]: dffac360ac20aabb4ccf2443b0f04daa47db3a0184c9855e1471b55dfb0fd69c 2024-06-21T01:56:41.228 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:40 smithi173 systemd[1]: Started Ceph osd.6 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:56:41.676 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:41 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T01:56:41.391+0000 7fb82c95f640 -1 Falling back to public interface 2024-06-21T01:56:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:41 smithi173 ceph-mon[57540]: pgmap v170: 161 pgs: 16 active+undersized, 21 peering, 1 stale+active+clean, 12 active+undersized+degraded, 111 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 45/723 objects degraded (6.224%) 2024-06-21T01:56:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:42.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:41 smithi092 ceph-mon[64276]: pgmap v170: 161 pgs: 16 active+undersized, 21 peering, 1 stale+active+clean, 12 active+undersized+degraded, 111 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 45/723 objects degraded (6.224%) 2024-06-21T01:56:42.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:42.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:41 smithi092 ceph-mon[66756]: pgmap v170: 161 pgs: 16 active+undersized, 21 peering, 1 stale+active+clean, 12 active+undersized+degraded, 111 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 45/723 objects degraded (6.224%) 2024-06-21T01:56:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:43.300 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:43 smithi173 ceph-mon[57540]: pgmap v171: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 86/723 objects degraded (11.895%) 2024-06-21T01:56:43.300 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:43 smithi173 ceph-mon[57540]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-21T01:56:43.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:43 smithi092 ceph-mon[64276]: pgmap v171: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 86/723 objects degraded (11.895%) 2024-06-21T01:56:43.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:43 smithi092 ceph-mon[64276]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-21T01:56:43.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:43 smithi092 ceph-mon[66756]: pgmap v171: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 86/723 objects degraded (11.895%) 2024-06-21T01:56:43.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:43 smithi092 ceph-mon[66756]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-21T01:56:44.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:44 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:44.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:44 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:44.472 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:44 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:44.473 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:44 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T01:56:44.114+0000 7fb82c95f640 -1 osd.6 0 read_superblock omap replica is missing. 2024-06-21T01:56:44.473 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:44 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T01:56:44.136+0000 7fb82c95f640 -1 osd.6 136 log_to_monitors true 2024-06-21T01:56:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:45 smithi092 ceph-mon[64276]: from='osd.6 [v2:172.21.15.173:6816/288365895,v1:172.21.15.173:6817/288365895]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:56:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:45 smithi092 ceph-mon[64276]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:56:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:45 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 86/723 objects degraded (11.895%), 20 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:45.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:45 smithi092 ceph-mon[64276]: pgmap v172: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 86/723 objects degraded (11.895%) 2024-06-21T01:56:45.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:45 smithi092 ceph-mon[66756]: from='osd.6 [v2:172.21.15.173:6816/288365895,v1:172.21.15.173:6817/288365895]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:56:45.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:45 smithi092 ceph-mon[66756]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:56:45.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:45 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 86/723 objects degraded (11.895%), 20 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:45.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:45 smithi092 ceph-mon[66756]: pgmap v172: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 86/723 objects degraded (11.895%) 2024-06-21T01:56:45.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:45 smithi173 ceph-mon[57540]: from='osd.6 [v2:172.21.15.173:6816/288365895,v1:172.21.15.173:6817/288365895]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:56:45.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:45 smithi173 ceph-mon[57540]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-21T01:56:45.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:45 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 86/723 objects degraded (11.895%), 20 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:45.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:45 smithi173 ceph-mon[57540]: pgmap v172: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 86/723 objects degraded (11.895%) 2024-06-21T01:56:45.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 01:56:45 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T01:56:45.484+0000 7fb820702640 -1 osd.6 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:56:46.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:46 smithi173 ceph-mon[57540]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-21T01:56:46.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:46 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:46.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:46 smithi173 ceph-mon[57540]: osdmap e139: 8 total, 7 up, 8 in 2024-06-21T01:56:46.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:46 smithi173 ceph-mon[57540]: from='osd.6 [v2:172.21.15.173:6816/288365895,v1:172.21.15.173:6817/288365895]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:46.338 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:46 smithi173 ceph-mon[57540]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:46.338 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:46 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:46.338 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:46 smithi173 ceph-mon[57540]: from='osd.6 ' entity='osd.6' 2024-06-21T01:56:46.338 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:46 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:46.338 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:46 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[64276]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-21T01:56:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:46.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[64276]: osdmap e139: 8 total, 7 up, 8 in 2024-06-21T01:56:46.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[64276]: from='osd.6 [v2:172.21.15.173:6816/288365895,v1:172.21.15.173:6817/288365895]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:46.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[64276]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:46.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:46.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[64276]: from='osd.6 ' entity='osd.6' 2024-06-21T01:56:46.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:46.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:46.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[66756]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-21T01:56:46.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:46.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[66756]: osdmap e139: 8 total, 7 up, 8 in 2024-06-21T01:56:46.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[66756]: from='osd.6 [v2:172.21.15.173:6816/288365895,v1:172.21.15.173:6817/288365895]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:46.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[66756]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:56:46.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:46.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[66756]: from='osd.6 ' entity='osd.6' 2024-06-21T01:56:46.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:46.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:46 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:47 smithi092 ceph-mon[64276]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:56:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:47 smithi092 ceph-mon[64276]: osd.6 [v2:172.21.15.173:6816/288365895,v1:172.21.15.173:6817/288365895] boot 2024-06-21T01:56:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:47 smithi092 ceph-mon[64276]: osdmap e140: 8 total, 8 up, 8 in 2024-06-21T01:56:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:47 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:56:47.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:47 smithi092 ceph-mon[64276]: pgmap v175: 161 pgs: 9 peering, 27 active+undersized, 17 active+undersized+degraded, 108 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 70/723 objects degraded (9.682%) 2024-06-21T01:56:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:47 smithi092 ceph-mon[66756]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:56:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:47 smithi092 ceph-mon[66756]: osd.6 [v2:172.21.15.173:6816/288365895,v1:172.21.15.173:6817/288365895] boot 2024-06-21T01:56:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:47 smithi092 ceph-mon[66756]: osdmap e140: 8 total, 8 up, 8 in 2024-06-21T01:56:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:47 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:56:47.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:47 smithi092 ceph-mon[66756]: pgmap v175: 161 pgs: 9 peering, 27 active+undersized, 17 active+undersized+degraded, 108 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 70/723 objects degraded (9.682%) 2024-06-21T01:56:47.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:47 smithi173 ceph-mon[57540]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:56:47.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:47 smithi173 ceph-mon[57540]: osd.6 [v2:172.21.15.173:6816/288365895,v1:172.21.15.173:6817/288365895] boot 2024-06-21T01:56:47.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:47 smithi173 ceph-mon[57540]: osdmap e140: 8 total, 8 up, 8 in 2024-06-21T01:56:47.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:47 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-21T01:56:47.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:47 smithi173 ceph-mon[57540]: pgmap v175: 161 pgs: 9 peering, 27 active+undersized, 17 active+undersized+degraded, 108 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 70/723 objects degraded (9.682%) 2024-06-21T01:56:48.598 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:48 smithi173 ceph-mon[57540]: osdmap e141: 8 total, 8 up, 8 in 2024-06-21T01:56:48.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:48 smithi092 ceph-mon[64276]: osdmap e141: 8 total, 8 up, 8 in 2024-06-21T01:56:48.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:48 smithi092 ceph-mon[66756]: osdmap e141: 8 total, 8 up, 8 in 2024-06-21T01:56:49.493 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:56:49.209+0000 7f9965fe7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-06-21T01:56:49.493 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:56:49] "GET /metrics HTTP/1.1" 200 38518 "" "Prometheus/2.33.4" 2024-06-21T01:56:49.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: pgmap v177: 161 pgs: 19 peering, 15 active+undersized, 10 active+undersized+degraded, 117 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 44/723 objects degraded (6.086%) 2024-06-21T01:56:49.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:49.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:49.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:49.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:56:49.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:49.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:56:49.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:49.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:49.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:49.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:56:49.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 70/723 objects degraded (9.682%), 17 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:49.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: pgmap v177: 161 pgs: 19 peering, 15 active+undersized, 10 active+undersized+degraded, 117 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 44/723 objects degraded (6.086%) 2024-06-21T01:56:49.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:49.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:49.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:49.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:56:49.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:49.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:56:49.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:49.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:49.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:49.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:56:49.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:49 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 70/723 objects degraded (9.682%), 17 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:49.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: pgmap v177: 161 pgs: 19 peering, 15 active+undersized, 10 active+undersized+degraded, 117 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 44/723 objects degraded (6.086%) 2024-06-21T01:56:49.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:49.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:49.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:56:49.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:56:49.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:49.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:56:49.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:49.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:49.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:56:49.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:56:49.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:49 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 70/723 objects degraded (9.682%), 17 pgs degraded (PG_DEGRADED) 2024-06-21T01:56:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:50 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:56:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:50 smithi092 ceph-mon[64276]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-06-21T01:56:50.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:50 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:56:50.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:50 smithi092 ceph-mon[66756]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-06-21T01:56:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:50 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:56:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:50 smithi173 ceph-mon[57540]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-06-21T01:56:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:51 smithi092 ceph-mon[64276]: pgmap v178: 161 pgs: 19 peering, 2 active+undersized, 1 active+undersized+degraded, 139 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 11/723 objects degraded (1.521%) 2024-06-21T01:56:51.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:51 smithi092 ceph-mon[66756]: pgmap v178: 161 pgs: 19 peering, 2 active+undersized, 1 active+undersized+degraded, 139 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 11/723 objects degraded (1.521%) 2024-06-21T01:56:51.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:51 smithi173 ceph-mon[57540]: pgmap v178: 161 pgs: 19 peering, 2 active+undersized, 1 active+undersized+degraded, 139 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 11/723 objects degraded (1.521%) 2024-06-21T01:56:53.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:53 smithi092 ceph-mon[64276]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:56:53.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:53 smithi092 ceph-mon[64276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/723 objects degraded (1.521%), 1 pg degraded) 2024-06-21T01:56:53.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:53 smithi092 ceph-mon[64276]: Cluster is now healthy 2024-06-21T01:56:53.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:53 smithi092 ceph-mon[66756]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:56:53.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:53 smithi092 ceph-mon[66756]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/723 objects degraded (1.521%), 1 pg degraded) 2024-06-21T01:56:53.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:53 smithi092 ceph-mon[66756]: Cluster is now healthy 2024-06-21T01:56:53.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:53 smithi173 ceph-mon[57540]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:56:53.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:53 smithi173 ceph-mon[57540]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/723 objects degraded (1.521%), 1 pg degraded) 2024-06-21T01:56:53.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:53 smithi173 ceph-mon[57540]: Cluster is now healthy 2024-06-21T01:56:54.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:54 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:54.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:54 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:54.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:54 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:56:55.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:55 smithi092 ceph-mon[64276]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:56:55.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:55 smithi092 ceph-mon[66756]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:56:55.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:55 smithi173 ceph-mon[57540]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:56:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:57.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:57.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:57.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:56:57.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:56:58.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:57 smithi173 ceph-mon[57540]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-06-21T01:56:58.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:57 smithi092 ceph-mon[64276]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-06-21T01:56:58.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:57 smithi092 ceph-mon[66756]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-06-21T01:56:59.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:56:59 smithi092 ceph-mon[64276]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 893 B/s rd, 0 op/s 2024-06-21T01:56:59.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:56:59 smithi092 ceph-mon[66756]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 893 B/s rd, 0 op/s 2024-06-21T01:56:59.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:56:59 smithi173 ceph-mon[57540]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 893 B/s rd, 0 op/s 2024-06-21T01:56:59.600 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:56:59 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:56:59] "GET /metrics HTTP/1.1" 200 38525 "" "Prometheus/2.33.4" 2024-06-21T01:57:01.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:01 smithi092 ceph-mon[64276]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:57:01.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:01 smithi092 ceph-mon[66756]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:57:01.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:01 smithi173 ceph-mon[57540]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:57:01.592 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:57:02.242 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (6m) 115s ago 10m 21.0M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (5m) 17s ago 9m 56.0M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (5m) 115s ago 9m 52.6M - 3.5 e1d6a67b021e 3192bc59456b 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (5m) 17s ago 14m 441M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (5m) 115s ago 16m 502M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (4m) 115s ago 16m 49.7M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (4m) 17s ago 15m 40.9M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (4m) 115s ago 15m 42.0M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (6m) 115s ago 11m 16.8M - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (5m) 17s ago 11m 20.6M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (3m) 115s ago 14m 52.6M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 7c252b0f2dfd 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (3m) 115s ago 14m 44.8M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 9ad8cda61390 2024-06-21T01:57:02.243 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (2m) 115s ago 13m 49.8M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 0fe024e7dad9 2024-06-21T01:57:02.244 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (119s) 115s ago 13m 14.5M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 f0817e4dfec7 2024-06-21T01:57:02.244 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (86s) 17s ago 13m 46.6M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 d42a6ffd2e66 2024-06-21T01:57:02.244 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (53s) 17s ago 12m 52.0M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 b6050a92252e 2024-06-21T01:57:02.244 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (21s) 17s ago 12m 12.1M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 dffac360ac20 2024-06-21T01:57:02.244 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (12m) 17s ago 12m 60.4M 2992M 17.2.0 e1d6a67b021e 91298552d0a4 2024-06-21T01:57:02.244 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (5m) 17s ago 11m 75.1M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:57:02.244 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (9m) 115s ago 9m 76.8M - 17.2.0 e1d6a67b021e 77ea4624d38b 2024-06-21T01:57:02.244 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (9m) 17s ago 9m 73.5M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:57:02.244 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (9m) 115s ago 9m 74.5M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:57:02.244 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (9m) 17s ago 9m 74.8M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:57:02.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:02 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:02.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:02 smithi092 ceph-mon[66756]: from='client.44422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:02.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:02 smithi092 ceph-mon[66756]: from='client.44428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:02.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:02 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:02.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:02 smithi092 ceph-mon[64276]: from='client.44422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:02.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:02 smithi092 ceph-mon[64276]: from='client.44428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:02.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:02 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:02.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:02 smithi173 ceph-mon[57540]: from='client.44422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:02.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:02 smithi173 ceph-mon[57540]: from='client.44428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:02.633 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:57:02.633 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:57:02.633 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 7 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:57:02.634 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-06-21T01:57:02.635 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 12 2024-06-21T01:57:02.635 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:57:02.635 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr", 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout: "mon" 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "12/23 daemons upgraded", 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading osd daemons", 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:57:02.977 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:57:03.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:03 smithi092 ceph-mon[64276]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:03.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:03 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/2377064066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:03.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:03 smithi092 ceph-mon[64276]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:57:03.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:03 smithi092 ceph-mon[66756]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:03.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:03 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/2377064066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:03.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:03 smithi092 ceph-mon[66756]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:57:03.375 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:57:03.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:03 smithi173 ceph-mon[57540]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:03.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:03 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/2377064066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:03.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:03 smithi173 ceph-mon[57540]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:57:04.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:04 smithi092 ceph-mon[64276]: from='client.44446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:04.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:04 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/3497586283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:57:04.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:04 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:04.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:04 smithi092 ceph-mon[66756]: from='client.44446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:04.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:04 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/3497586283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:57:04.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:04 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:04.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:04 smithi173 ceph-mon[57540]: from='client.44446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:04.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:04 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/3497586283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:57:04.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:04 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:05.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:05 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:57:05.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:05 smithi173 ceph-mon[57540]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:57:05.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:05 smithi173 ceph-mon[57540]: Upgrade: osd.7 is safe to restart 2024-06-21T01:57:05.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:05 smithi173 ceph-mon[57540]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:57:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:05 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:57:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:05 smithi092 ceph-mon[64276]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:57:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:05 smithi092 ceph-mon[64276]: Upgrade: osd.7 is safe to restart 2024-06-21T01:57:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:05 smithi092 ceph-mon[64276]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:57:05.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:05 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:57:05.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:05 smithi092 ceph-mon[66756]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-21T01:57:05.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:05 smithi092 ceph-mon[66756]: Upgrade: osd.7 is safe to restart 2024-06-21T01:57:05.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:05 smithi092 ceph-mon[66756]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:57:06.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:06 smithi173 ceph-mon[57540]: Upgrade: Updating osd.7 2024-06-21T01:57:06.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:06.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-21T01:57:06.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:06 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:06.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:06 smithi173 ceph-mon[57540]: Deploying daemon osd.7 on smithi173 2024-06-21T01:57:07.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:06 smithi092 ceph-mon[64276]: Upgrade: Updating osd.7 2024-06-21T01:57:07.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:07.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-21T01:57:07.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:06 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:07.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:06 smithi092 ceph-mon[64276]: Deploying daemon osd.7 on smithi173 2024-06-21T01:57:07.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:06 smithi092 ceph-mon[66756]: Upgrade: Updating osd.7 2024-06-21T01:57:07.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:07.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-21T01:57:07.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:06 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:07.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:06 smithi092 ceph-mon[66756]: Deploying daemon osd.7 on smithi173 2024-06-21T01:57:07.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:07 smithi173 ceph-mon[57540]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:57:08.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:07 smithi092 ceph-mon[64276]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:57:08.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:07 smithi092 ceph-mon[66756]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:57:09.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:09 smithi173 ceph-mon[57540]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:57:09.020 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:08 smithi173 systemd[1]: Stopping Ceph osd.7 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:57:09.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:09 smithi092 ceph-mon[64276]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:57:09.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:09 smithi092 ceph-mon[66756]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:57:09.478 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[38091]: 2024-06-21T01:57:09.044+0000 7fcaf972d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T01:57:09.478 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[38091]: 2024-06-21T01:57:09.044+0000 7fcaf972d700 -1 osd.7 141 *** Got signal Terminated *** 2024-06-21T01:57:09.478 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[38091]: 2024-06-21T01:57:09.044+0000 7fcaf972d700 -1 osd.7 141 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T01:57:09.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:57:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:57:09] "GET /metrics HTTP/1.1" 200 38525 "" "Prometheus/2.33.4" 2024-06-21T01:57:10.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:10 smithi173 ceph-mon[57540]: osd.7 marked itself down and dead 2024-06-21T01:57:10.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:10 smithi092 ceph-mon[64276]: osd.7 marked itself down and dead 2024-06-21T01:57:10.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:10 smithi092 ceph-mon[66756]: osd.7 marked itself down and dead 2024-06-21T01:57:10.718 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:10 smithi173 podman[67195]: 2024-06-21 01:57:10.471226532 +0000 UTC m=+1.525150325 container died 91298552d0a41fed97ade9f89a225cc09e7cae4fe70be6c75606c063a3a51cbd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7, architecture=x86_64, distribution-scope=public, maintainer=Guillaume Abrioux , name=centos-stream, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, version=8, RELEASE=HEAD, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream) 2024-06-21T01:57:10.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:10 smithi173 podman[67195]: 2024-06-21 01:57:10.787043704 +0000 UTC m=+1.840967497 container cleanup 91298552d0a41fed97ade9f89a225cc09e7cae4fe70be6c75606c063a3a51cbd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, distribution-scope=public, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., RELEASE=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, release=754) 2024-06-21T01:57:10.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:10 smithi173 bash[67195]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7 2024-06-21T01:57:10.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:10 smithi173 podman[67223]: 2024-06-21 01:57:10.954884682 +0000 UTC m=+0.477854783 container remove 91298552d0a41fed97ade9f89a225cc09e7cae4fe70be6c75606c063a3a51cbd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7, release=754, RELEASE=HEAD, GIT_CLEAN=True, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.buildah.version=1.19.8, name=centos-stream, vendor=Red Hat, Inc., vcs-type=git, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0) 2024-06-21T01:57:11.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:57:11.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-06-21T01:57:11.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: osdmap e142: 8 total, 7 up, 8 in 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 7]}]: dispatch 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 5]}]: dispatch 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]: dispatch 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]: dispatch 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [4, 7]}]: dispatch 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]: dispatch 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 7]}]: dispatch 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: pgmap v189: 161 pgs: 22 peering, 11 stale+active+clean, 128 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 7]}]': finished 2024-06-21T01:57:11.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 5]}]': finished 2024-06-21T01:57:11.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]': finished 2024-06-21T01:57:11.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]': finished 2024-06-21T01:57:11.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-06-21T01:57:11.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [4, 7]}]': finished 2024-06-21T01:57:11.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-06-21T01:57:11.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:57:11.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-06-21T01:57:11.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: osdmap e142: 8 total, 7 up, 8 in 2024-06-21T01:57:11.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 7]}]: dispatch 2024-06-21T01:57:11.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 5]}]: dispatch 2024-06-21T01:57:11.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]: dispatch 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]: dispatch 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [4, 7]}]: dispatch 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]: dispatch 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 7]}]: dispatch 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: pgmap v189: 161 pgs: 22 peering, 11 stale+active+clean, 128 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 7]}]': finished 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 5]}]': finished 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]': finished 2024-06-21T01:57:11.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]': finished 2024-06-21T01:57:11.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-06-21T01:57:11.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [4, 7]}]': finished 2024-06-21T01:57:11.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-06-21T01:57:11.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]': finished 2024-06-21T01:57:11.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 7]}]': finished 2024-06-21T01:57:11.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[66756]: osdmap e143: 8 total, 7 up, 8 in 2024-06-21T01:57:11.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]': finished 2024-06-21T01:57:11.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 7]}]': finished 2024-06-21T01:57:11.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:11 smithi092 ceph-mon[64276]: osdmap e143: 8 total, 7 up, 8 in 2024-06-21T01:57:11.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-21T01:57:11.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: osdmap e142: 8 total, 7 up, 8 in 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 7]}]: dispatch 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 5]}]: dispatch 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]: dispatch 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]: dispatch 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]: dispatch 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [4, 7]}]: dispatch 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]: dispatch 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 7]}]: dispatch 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:11.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: pgmap v189: 161 pgs: 22 peering, 11 stale+active+clean, 128 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-06-21T01:57:11.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 7]}]': finished 2024-06-21T01:57:11.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 5]}]': finished 2024-06-21T01:57:11.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]': finished 2024-06-21T01:57:11.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 0]}]': finished 2024-06-21T01:57:11.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-06-21T01:57:11.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [4, 7]}]': finished 2024-06-21T01:57:11.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-06-21T01:57:11.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 7]}]': finished 2024-06-21T01:57:11.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 7]}]': finished 2024-06-21T01:57:11.369 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:11 smithi173 ceph-mon[57540]: osdmap e143: 8 total, 7 up, 8 in 2024-06-21T01:57:11.654 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:11 smithi173 podman[67311]: 2024-06-21 01:57:11.273861651 +0000 UTC m=+0.017280203 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:57:11.655 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:11 smithi173 podman[67311]: 2024-06-21 01:57:11.399080944 +0000 UTC m=+0.142499487 container create fb97da41d2f3ae04cb961b3d7cd2984a38c912e38af3ee82b8223921ab33ad99 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, ceph=True, org.label-schema.schema-version=1.0) 2024-06-21T01:57:11.655 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:11 smithi173 podman[67311]: 2024-06-21 01:57:11.658849656 +0000 UTC m=+0.402268208 container init fb97da41d2f3ae04cb961b3d7cd2984a38c912e38af3ee82b8223921ab33ad99 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:57:11.911 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:11 smithi173 podman[67311]: 2024-06-21 01:57:11.662288611 +0000 UTC m=+0.405707159 container start fb97da41d2f3ae04cb961b3d7cd2984a38c912e38af3ee82b8223921ab33ad99 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:57:11.911 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:11 smithi173 podman[67311]: 2024-06-21 01:57:11.733776284 +0000 UTC m=+0.477194843 container attach fb97da41d2f3ae04cb961b3d7cd2984a38c912e38af3ee82b8223921ab33ad99 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-deactivate, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True) 2024-06-21T01:57:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:12 smithi173 ceph-mon[57540]: Health check failed: Reduced data availability: 1 pg inactive, 5 pgs peering (PG_AVAILABILITY) 2024-06-21T01:57:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:12 smithi173 ceph-mon[57540]: osdmap e144: 8 total, 7 up, 8 in 2024-06-21T01:57:12.228 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:11 smithi173 podman[67311]: 2024-06-21 01:57:11.914870117 +0000 UTC m=+0.658288662 container died fb97da41d2f3ae04cb961b3d7cd2984a38c912e38af3ee82b8223921ab33ad99 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True) 2024-06-21T01:57:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:12 smithi092 ceph-mon[64276]: Health check failed: Reduced data availability: 1 pg inactive, 5 pgs peering (PG_AVAILABILITY) 2024-06-21T01:57:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:12 smithi092 ceph-mon[64276]: osdmap e144: 8 total, 7 up, 8 in 2024-06-21T01:57:12.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:12 smithi092 ceph-mon[66756]: Health check failed: Reduced data availability: 1 pg inactive, 5 pgs peering (PG_AVAILABILITY) 2024-06-21T01:57:12.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:12 smithi092 ceph-mon[66756]: osdmap e144: 8 total, 7 up, 8 in 2024-06-21T01:57:12.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:12 smithi173 podman[67334]: 2024-06-21 01:57:12.554187882 +0000 UTC m=+0.633309694 container remove fb97da41d2f3ae04cb961b3d7cd2984a38c912e38af3ee82b8223921ab33ad99 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618) 2024-06-21T01:57:12.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:12 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.7.service: Deactivated successfully. 2024-06-21T01:57:12.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:12 smithi173 systemd[1]: Stopped Ceph osd.7 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:57:12.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:12 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.7.service: Consumed 9.817s CPU time. 2024-06-21T01:57:12.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:12 smithi173 systemd[1]: Starting Ceph osd.7 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T01:57:13.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:13 smithi173 ceph-mon[57540]: pgmap v192: 161 pgs: 8 active+undersized, 27 peering, 9 stale+active+clean, 5 active+undersized+degraded, 112 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 22/723 objects degraded (3.043%) 2024-06-21T01:57:13.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:13 smithi173 ceph-mon[57540]: osdmap e145: 8 total, 7 up, 8 in 2024-06-21T01:57:13.246 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:12 smithi173 podman[67436]: 2024-06-21 01:57:12.898525408 +0000 UTC m=+0.017582089 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:57:13.246 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 podman[67436]: 2024-06-21 01:57:13.025896007 +0000 UTC m=+0.144952692 container create 5874bd6abb397c34f70e7e11645822fe2ad542d6e4c28f4605547d10c28e2e09 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True) 2024-06-21T01:57:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:13 smithi092 ceph-mon[64276]: pgmap v192: 161 pgs: 8 active+undersized, 27 peering, 9 stale+active+clean, 5 active+undersized+degraded, 112 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 22/723 objects degraded (3.043%) 2024-06-21T01:57:13.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:13 smithi092 ceph-mon[64276]: osdmap e145: 8 total, 7 up, 8 in 2024-06-21T01:57:13.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:13 smithi092 ceph-mon[66756]: pgmap v192: 161 pgs: 8 active+undersized, 27 peering, 9 stale+active+clean, 5 active+undersized+degraded, 112 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 22/723 objects degraded (3.043%) 2024-06-21T01:57:13.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:13 smithi092 ceph-mon[66756]: osdmap e145: 8 total, 7 up, 8 in 2024-06-21T01:57:13.610 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 podman[67436]: 2024-06-21 01:57:13.250369879 +0000 UTC m=+0.369426557 container init 5874bd6abb397c34f70e7e11645822fe2ad542d6e4c28f4605547d10c28e2e09 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-06-21T01:57:13.610 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 podman[67436]: 2024-06-21 01:57:13.253911497 +0000 UTC m=+0.372968181 container start 5874bd6abb397c34f70e7e11645822fe2ad542d6e4c28f4605547d10c28e2e09 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-06-21T01:57:13.611 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 podman[67436]: 2024-06-21 01:57:13.333315807 +0000 UTC m=+0.452372492 container attach 5874bd6abb397c34f70e7e11645822fe2ad542d6e4c28f4605547d10c28e2e09 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-21T01:57:13.974 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate[67451]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 bash[67436]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate[67451]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 bash[67436]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate[67451]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 bash[67436]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate[67451]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 bash[67436]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate[67451]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 bash[67436]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate[67451]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 bash[67436]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate[67451]: --> ceph-volume raw activate successful for osd ID: 7 2024-06-21T01:57:13.975 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 bash[67436]: --> ceph-volume raw activate successful for osd ID: 7 2024-06-21T01:57:13.976 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:13 smithi173 podman[67436]: 2024-06-21 01:57:13.667701407 +0000 UTC m=+0.786758084 container died 5874bd6abb397c34f70e7e11645822fe2ad542d6e4c28f4605547d10c28e2e09 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-21T01:57:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:14 smithi173 ceph-mon[57540]: Health check failed: Degraded data redundancy: 22/723 objects degraded (3.043%), 5 pgs degraded (PG_DEGRADED) 2024-06-21T01:57:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:14 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:14.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:14 smithi092 ceph-mon[64276]: Health check failed: Degraded data redundancy: 22/723 objects degraded (3.043%), 5 pgs degraded (PG_DEGRADED) 2024-06-21T01:57:14.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:14 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:14.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:14 smithi092 ceph-mon[66756]: Health check failed: Degraded data redundancy: 22/723 objects degraded (3.043%), 5 pgs degraded (PG_DEGRADED) 2024-06-21T01:57:14.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:14 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:14.508 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:14 smithi173 podman[67574]: 2024-06-21 01:57:14.26188013 +0000 UTC m=+0.587940329 container remove 5874bd6abb397c34f70e7e11645822fe2ad542d6e4c28f4605547d10c28e2e09 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240618) 2024-06-21T01:57:14.508 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:14 smithi173 podman[67634]: 2024-06-21 01:57:14.381344619 +0000 UTC m=+0.017526607 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T01:57:14.820 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:14 smithi173 podman[67634]: 2024-06-21 01:57:14.512838893 +0000 UTC m=+0.149020884 container create 3d835c4a13e683d9933f6eebc80045c8e97f91e81fe7d2f2358a54e9616ccbdd (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, GIT_CLEAN=True) 2024-06-21T01:57:14.820 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:14 smithi173 podman[67634]: 2024-06-21 01:57:14.732149236 +0000 UTC m=+0.368331224 container init 3d835c4a13e683d9933f6eebc80045c8e97f91e81fe7d2f2358a54e9616ccbdd (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T01:57:14.820 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:14 smithi173 podman[67634]: 2024-06-21 01:57:14.735430199 +0000 UTC m=+0.371612188 container start 3d835c4a13e683d9933f6eebc80045c8e97f91e81fe7d2f2358a54e9616ccbdd (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T01:57:14.820 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:14 smithi173 ceph-osd[67653]: -- 172.21.15.173:0/2669828444 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5577977c5a40 con 0x557798584400 2024-06-21T01:57:15.228 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:14 smithi173 bash[67634]: 3d835c4a13e683d9933f6eebc80045c8e97f91e81fe7d2f2358a54e9616ccbdd 2024-06-21T01:57:15.228 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:14 smithi173 systemd[1]: Started Ceph osd.7 for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T01:57:15.640 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T01:57:15.283+0000 7f496df1e640 -1 Falling back to public interface 2024-06-21T01:57:15.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:15.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:15 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:15.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:15 smithi173 ceph-mon[57540]: pgmap v194: 161 pgs: 3 active+clean+remapped, 2 remapped+peering, 28 active+undersized, 27 peering, 13 active+undersized+degraded, 88 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 49/723 objects degraded (6.777%) 2024-06-21T01:57:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:15 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:16.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:15 smithi092 ceph-mon[64276]: pgmap v194: 161 pgs: 3 active+clean+remapped, 2 remapped+peering, 28 active+undersized, 27 peering, 13 active+undersized+degraded, 88 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 49/723 objects degraded (6.777%) 2024-06-21T01:57:16.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:16.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:15 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:16.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:15 smithi092 ceph-mon[66756]: pgmap v194: 161 pgs: 3 active+clean+remapped, 2 remapped+peering, 28 active+undersized, 27 peering, 13 active+undersized+degraded, 88 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 49/723 objects degraded (6.777%) 2024-06-21T01:57:17.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:17 smithi173 ceph-mon[57540]: pgmap v195: 161 pgs: 5 active+clean+remapped, 2 remapped+peering, 45 active+undersized, 22 active+undersized+degraded, 87 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 90/723 objects degraded (12.448%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:17.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:17 smithi173 ceph-mon[57540]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-21T01:57:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:17 smithi092 ceph-mon[64276]: pgmap v195: 161 pgs: 5 active+clean+remapped, 2 remapped+peering, 45 active+undersized, 22 active+undersized+degraded, 87 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 90/723 objects degraded (12.448%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:17.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:17 smithi092 ceph-mon[64276]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-21T01:57:17.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:17 smithi092 ceph-mon[66756]: pgmap v195: 161 pgs: 5 active+clean+remapped, 2 remapped+peering, 45 active+undersized, 22 active+undersized+degraded, 87 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 90/723 objects degraded (12.448%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:17.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:17 smithi092 ceph-mon[66756]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-21T01:57:18.042 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:18 smithi173 ceph-mon[57540]: from='osd.7 [v2:172.21.15.173:6824/119420962,v1:172.21.15.173:6825/119420962]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:57:18.042 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:18 smithi173 ceph-mon[57540]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:57:18.042 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:17 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T01:57:17.980+0000 7f496df1e640 -1 osd.7 0 read_superblock omap replica is missing. 2024-06-21T01:57:18.042 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T01:57:18.006+0000 7f496df1e640 -1 osd.7 141 log_to_monitors true 2024-06-21T01:57:18.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:18 smithi092 ceph-mon[64276]: from='osd.7 [v2:172.21.15.173:6824/119420962,v1:172.21.15.173:6825/119420962]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:57:18.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:18 smithi092 ceph-mon[64276]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:57:18.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:18 smithi092 ceph-mon[66756]: from='osd.7 [v2:172.21.15.173:6824/119420962,v1:172.21.15.173:6825/119420962]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:57:18.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:18 smithi092 ceph-mon[66756]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-21T01:57:18.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 01:57:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T01:57:18.487+0000 7f4961cc1640 -1 osd.7 141 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-21T01:57:19.272 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[64276]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-21T01:57:19.273 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[64276]: from='osd.7 [v2:172.21.15.173:6824/119420962,v1:172.21.15.173:6825/119420962]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:57:19.273 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[64276]: osdmap e146: 8 total, 7 up, 8 in 2024-06-21T01:57:19.273 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[64276]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:57:19.273 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[64276]: from='osd.7 ' entity='osd.7' 2024-06-21T01:57:19.273 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:19.273 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:19.273 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[64276]: pgmap v197: 161 pgs: 7 active+clean+remapped, 45 active+undersized, 22 active+undersized+degraded, 87 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 90/723 objects degraded (12.448%); 1/723 objects misplaced (0.138%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:19.273 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[66756]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-21T01:57:19.273 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[66756]: from='osd.7 [v2:172.21.15.173:6824/119420962,v1:172.21.15.173:6825/119420962]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:57:19.273 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[66756]: osdmap e146: 8 total, 7 up, 8 in 2024-06-21T01:57:19.273 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[66756]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:57:19.274 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[66756]: from='osd.7 ' entity='osd.7' 2024-06-21T01:57:19.274 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:19.274 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:19.274 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-mon[66756]: pgmap v197: 161 pgs: 7 active+clean+remapped, 45 active+undersized, 22 active+undersized+degraded, 87 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 90/723 objects degraded (12.448%); 1/723 objects misplaced (0.138%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:19.429 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:19 smithi173 ceph-mon[57540]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-21T01:57:19.430 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:19 smithi173 ceph-mon[57540]: from='osd.7 [v2:172.21.15.173:6824/119420962,v1:172.21.15.173:6825/119420962]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:57:19.430 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:19 smithi173 ceph-mon[57540]: osdmap e146: 8 total, 7 up, 8 in 2024-06-21T01:57:19.430 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:19 smithi173 ceph-mon[57540]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-21T01:57:19.430 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:19 smithi173 ceph-mon[57540]: from='osd.7 ' entity='osd.7' 2024-06-21T01:57:19.430 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:19 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:19.430 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:19 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:19.431 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:19 smithi173 ceph-mon[57540]: pgmap v197: 161 pgs: 7 active+clean+remapped, 45 active+undersized, 22 active+undersized+degraded, 87 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 90/723 objects degraded (12.448%); 1/723 objects misplaced (0.138%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:19.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:57:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:57:19] "GET /metrics HTTP/1.1" 200 38605 "" "Prometheus/2.33.4" 2024-06-21T01:57:20.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:20 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 90/723 objects degraded (12.448%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:57:20.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:20 smithi173 ceph-mon[57540]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:57:20.231 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:20 smithi173 ceph-mon[57540]: osd.7 [v2:172.21.15.173:6824/119420962,v1:172.21.15.173:6825/119420962] boot 2024-06-21T01:57:20.231 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:20 smithi173 ceph-mon[57540]: osdmap e147: 8 total, 8 up, 8 in 2024-06-21T01:57:20.231 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:20 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:57:20.231 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:20 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:20.231 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:20 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:20.231 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:20 smithi173 ceph-mon[57540]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-21T01:57:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 90/723 objects degraded (12.448%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:57:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[64276]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:57:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[64276]: osd.7 [v2:172.21.15.173:6824/119420962,v1:172.21.15.173:6825/119420962] boot 2024-06-21T01:57:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[64276]: osdmap e147: 8 total, 8 up, 8 in 2024-06-21T01:57:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:57:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:20.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:20.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[64276]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-21T01:57:20.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 90/723 objects degraded (12.448%), 22 pgs degraded (PG_DEGRADED) 2024-06-21T01:57:20.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[66756]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-21T01:57:20.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[66756]: osd.7 [v2:172.21.15.173:6824/119420962,v1:172.21.15.173:6825/119420962] boot 2024-06-21T01:57:20.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[66756]: osdmap e147: 8 total, 8 up, 8 in 2024-06-21T01:57:20.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-21T01:57:20.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:20.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:20.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:20 smithi092 ceph-mon[66756]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-21T01:57:21.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:21 smithi173 ceph-mon[57540]: osdmap e148: 8 total, 8 up, 8 in 2024-06-21T01:57:21.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:21 smithi173 ceph-mon[57540]: pgmap v200: 161 pgs: 2 remapped+peering, 9 peering, 5 active+clean+remapped, 39 active+undersized, 19 active+undersized+degraded, 87 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 4 op/s; 75/723 objects degraded (10.373%); 1/723 objects misplaced (0.138%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:21.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:21 smithi092 ceph-mon[64276]: osdmap e148: 8 total, 8 up, 8 in 2024-06-21T01:57:21.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:21 smithi092 ceph-mon[64276]: pgmap v200: 161 pgs: 2 remapped+peering, 9 peering, 5 active+clean+remapped, 39 active+undersized, 19 active+undersized+degraded, 87 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 4 op/s; 75/723 objects degraded (10.373%); 1/723 objects misplaced (0.138%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:21.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:21 smithi092 ceph-mon[66756]: osdmap e148: 8 total, 8 up, 8 in 2024-06-21T01:57:21.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:21 smithi092 ceph-mon[66756]: pgmap v200: 161 pgs: 2 remapped+peering, 9 peering, 5 active+clean+remapped, 39 active+undersized, 19 active+undersized+degraded, 87 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 0 B/s wr, 4 op/s; 75/723 objects degraded (10.373%); 1/723 objects misplaced (0.138%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:22.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:22 smithi173 ceph-mon[57540]: osdmap e149: 8 total, 8 up, 8 in 2024-06-21T01:57:22.850 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:22 smithi092 ceph-mon[64276]: osdmap e149: 8 total, 8 up, 8 in 2024-06-21T01:57:22.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:22 smithi092 ceph-mon[66756]: osdmap e149: 8 total, 8 up, 8 in 2024-06-21T01:57:24.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:24.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:57:24.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:57:24.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: Upgrade: Setting container_image for all osd 2024-06-21T01:57:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: pgmap v202: 161 pgs: 2 remapped+peering, 9 peering, 5 active+clean+remapped, 29 active+undersized, 13 active+undersized+degraded, 103 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s; 56/723 objects degraded (7.746%); 1/723 objects misplaced (0.138%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-06-21T01:57:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-06-21T01:57:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-06-21T01:57:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-06-21T01:57:24.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-06-21T01:57:24.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-06-21T01:57:24.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-06-21T01:57:24.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-06-21T01:57:24.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-06-21T01:57:24.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-06-21T01:57:24.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: Upgrade: Setting container_image for all osd 2024-06-21T01:57:24.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: pgmap v202: 161 pgs: 2 remapped+peering, 9 peering, 5 active+clean+remapped, 29 active+undersized, 13 active+undersized+degraded, 103 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s; 56/723 objects degraded (7.746%); 1/723 objects misplaced (0.138%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-06-21T01:57:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-06-21T01:57:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-06-21T01:57:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-06-21T01:57:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-06-21T01:57:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-06-21T01:57:24.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-06-21T01:57:24.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-06-21T01:57:24.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-06-21T01:57:24.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-06-21T01:57:24.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-06-21T01:57:24.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:23 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-06-21T01:57:24.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: Upgrade: Setting container_image for all osd 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: pgmap v202: 161 pgs: 2 remapped+peering, 9 peering, 5 active+clean+remapped, 29 active+undersized, 13 active+undersized+degraded, 103 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 68 op/s; 56/723 objects degraded (7.746%); 1/723 objects misplaced (0.138%); 0 B/s, 0 objects/s recovering 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-06-21T01:57:24.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-06-21T01:57:24.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-06-21T01:57:24.231 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:23 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-06-21T01:57:25.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:24 smithi173 ceph-mon[57540]: Upgrade: Setting require_osd_release to 19 squid 2024-06-21T01:57:25.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:24 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:25.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:24 smithi173 ceph-mon[57540]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-06-21T01:57:25.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:24 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-06-21T01:57:25.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:24 smithi173 ceph-mon[57540]: osdmap e150: 8 total, 8 up, 8 in 2024-06-21T01:57:25.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:24 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:25.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:24 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:25.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:24 smithi173 ceph-mon[57540]: Health check update: Degraded data redundancy: 56/723 objects degraded (7.746%), 13 pgs degraded (PG_DEGRADED) 2024-06-21T01:57:25.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[64276]: Upgrade: Setting require_osd_release to 19 squid 2024-06-21T01:57:25.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:25.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[64276]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-06-21T01:57:25.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-06-21T01:57:25.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[64276]: osdmap e150: 8 total, 8 up, 8 in 2024-06-21T01:57:25.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:25.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:25.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[64276]: Health check update: Degraded data redundancy: 56/723 objects degraded (7.746%), 13 pgs degraded (PG_DEGRADED) 2024-06-21T01:57:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[66756]: Upgrade: Setting require_osd_release to 19 squid 2024-06-21T01:57:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:25.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[66756]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-06-21T01:57:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-06-21T01:57:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[66756]: osdmap e150: 8 total, 8 up, 8 in 2024-06-21T01:57:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:25.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:24 smithi092 ceph-mon[66756]: Health check update: Degraded data redundancy: 56/723 objects degraded (7.746%), 13 pgs degraded (PG_DEGRADED) 2024-06-21T01:57:26.100 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[64276]: Upgrade: Setting container_image for all mds 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[64276]: pgmap v204: 161 pgs: 1 remapped+peering, 9 peering, 151 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 0 B/s wr, 198 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[64276]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/723 objects degraded (7.746%), 13 pgs degraded) 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[64276]: Cluster is now healthy 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[66756]: Upgrade: Setting container_image for all mds 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[66756]: pgmap v204: 161 pgs: 1 remapped+peering, 9 peering, 151 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 0 B/s wr, 198 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[66756]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/723 objects degraded (7.746%), 13 pgs degraded) 2024-06-21T01:57:26.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[66756]: Cluster is now healthy 2024-06-21T01:57:26.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:26.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:26.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:26.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:26.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:25 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:26.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:25 smithi173 ceph-mon[57540]: Upgrade: Setting container_image for all mds 2024-06-21T01:57:26.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:25 smithi173 ceph-mon[57540]: pgmap v204: 161 pgs: 1 remapped+peering, 9 peering, 151 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 0 B/s wr, 198 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:26.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:25 smithi173 ceph-mon[57540]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 56/723 objects degraded (7.746%), 13 pgs degraded) 2024-06-21T01:57:26.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:25 smithi173 ceph-mon[57540]: Cluster is now healthy 2024-06-21T01:57:26.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:25 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:26.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:25 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.lzbkvh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:26.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:25 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:26.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:25 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:26.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:25 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:26.990 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:26 smithi092 ceph-mon[64276]: Upgrade: Updating rgw.foo.smithi092.lzbkvh (1/4) 2024-06-21T01:57:26.990 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:26 smithi092 ceph-mon[64276]: Deploying daemon rgw.foo.smithi092.lzbkvh on smithi092 2024-06-21T01:57:26.990 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:26 smithi092 ceph-mon[66756]: Upgrade: Updating rgw.foo.smithi092.lzbkvh (1/4) 2024-06-21T01:57:26.990 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:26 smithi092 ceph-mon[66756]: Deploying daemon rgw.foo.smithi092.lzbkvh on smithi092 2024-06-21T01:57:27.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:26 smithi173 ceph-mon[57540]: Upgrade: Updating rgw.foo.smithi092.lzbkvh (1/4) 2024-06-21T01:57:27.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:26 smithi173 ceph-mon[57540]: Deploying daemon rgw.foo.smithi092.lzbkvh on smithi092 2024-06-21T01:57:28.227 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:27 smithi092 ceph-mon[64276]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 130 KiB/s rd, 0 B/s wr, 216 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:28.227 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:27 smithi092 ceph-mon[66756]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 130 KiB/s rd, 0 B/s wr, 216 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:27 smithi173 ceph-mon[57540]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 130 KiB/s rd, 0 B/s wr, 216 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:29.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:28 smithi173 ceph-mon[57540]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 0 B/s wr, 244 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:29.265 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:28 smithi092 ceph-mon[66756]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 0 B/s wr, 244 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:29.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:28 smithi092 ceph-mon[64276]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 0 B/s wr, 244 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:29.556 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:57:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:57:29] "GET /metrics HTTP/1.1" 200 38616 "" "Prometheus/2.33.4" 2024-06-21T01:57:30.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:30 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:30.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:30 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:30.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:30 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:30.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:30 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:30.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:30 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:30.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:30 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:32.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[66756]: Upgrade: Updating rgw.smpl.smithi092.bxxcjp (2/4) 2024-06-21T01:57:32.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:32.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:32.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:32.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[66756]: Deploying daemon rgw.smpl.smithi092.bxxcjp on smithi092 2024-06-21T01:57:32.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[66756]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s rd, 0 B/s wr, 245 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:32.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[64276]: Upgrade: Updating rgw.smpl.smithi092.bxxcjp (2/4) 2024-06-21T01:57:32.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:32.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:32.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:32.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[64276]: Deploying daemon rgw.smpl.smithi092.bxxcjp on smithi092 2024-06-21T01:57:32.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:31 smithi092 ceph-mon[64276]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s rd, 0 B/s wr, 245 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:31 smithi173 ceph-mon[57540]: Upgrade: Updating rgw.smpl.smithi092.bxxcjp (2/4) 2024-06-21T01:57:32.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:31 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:32.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:31 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.bxxcjp", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:32.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:31 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:32.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:31 smithi173 ceph-mon[57540]: Deploying daemon rgw.smpl.smithi092.bxxcjp on smithi092 2024-06-21T01:57:32.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:31 smithi173 ceph-mon[57540]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s rd, 0 B/s wr, 245 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:33.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:33 smithi173 ceph-mon[57540]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 102 B/s wr, 274 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:33.250 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:33 smithi092 ceph-mon[64276]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 102 B/s wr, 274 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:33.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:33 smithi092 ceph-mon[66756]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 102 B/s wr, 274 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:33.740 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:57:34.285 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:34 smithi092 ceph-mon[64276]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:34.285 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:34 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:34.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:34 smithi092 ceph-mon[66756]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:34.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:34 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:34.461 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (6m) 2m ago 10m 21.0M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (6m) 15s ago 10m 56.2M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (5m) 2m ago 10m 52.6M - 3.5 e1d6a67b021e 3192bc59456b 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (5m) 15s ago 15m 441M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (6m) 2m ago 17m 502M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (5m) 2m ago 17m 49.7M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (4m) 15s ago 15m 42.5M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (5m) 2m ago 15m 42.0M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (6m) 2m ago 11m 16.8M - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (6m) 15s ago 11m 17.6M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (4m) 2m ago 15m 52.6M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 7c252b0f2dfd 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (3m) 2m ago 14m 44.8M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 9ad8cda61390 2024-06-21T01:57:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (3m) 2m ago 14m 49.8M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 0fe024e7dad9 2024-06-21T01:57:34.463 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (2m) 2m ago 14m 14.5M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 f0817e4dfec7 2024-06-21T01:57:34.463 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (118s) 15s ago 13m 48.9M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 d42a6ffd2e66 2024-06-21T01:57:34.463 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (86s) 15s ago 13m 52.8M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 b6050a92252e 2024-06-21T01:57:34.463 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (53s) 15s ago 12m 51.5M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 dffac360ac20 2024-06-21T01:57:34.463 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (20s) 15s ago 12m 12.1M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 3d835c4a13e6 2024-06-21T01:57:34.463 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (5m) 15s ago 11m 75.1M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:57:34.463 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 starting - - - - 2024-06-21T01:57:34.463 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (10m) 15s ago 10m 73.7M - 17.2.0 e1d6a67b021e 58c5ea182e75 2024-06-21T01:57:34.463 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (10m) 2m ago 10m 74.5M - 17.2.0 e1d6a67b021e 112df87ddfce 2024-06-21T01:57:34.463 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (10m) 15s ago 10m 74.9M - 17.2.0 e1d6a67b021e 3f1474a534c6 2024-06-21T01:57:34.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:34 smithi173 ceph-mon[57540]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:34.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:34 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:34.984 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:57:34.984 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:57:34.984 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:57:34.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:57:34.984 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:57:34.984 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:57:34.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:57:34.984 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:57:34.984 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 8 2024-06-21T01:57:34.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:57:34.985 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:57:34.985 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-06-21T01:57:34.985 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 1 2024-06-21T01:57:34.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:57:34.985 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:57:34.985 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-06-21T01:57:34.985 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 14 2024-06-21T01:57:34.985 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:57:34.985 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:57:35.317 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:57:35.317 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:57:35.317 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:57:35.317 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:57:35.317 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:57:35.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd", 2024-06-21T01:57:35.318 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr", 2024-06-21T01:57:35.318 INFO:teuthology.orchestra.run.smithi092.stdout: "mon" 2024-06-21T01:57:35.318 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:57:35.318 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "13/23 daemons upgraded", 2024-06-21T01:57:35.318 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading rgw daemons", 2024-06-21T01:57:35.318 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:57:35.318 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:57:35.710 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:57:35.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:35 smithi173 ceph-mon[57540]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:35.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:35 smithi173 ceph-mon[57540]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:35.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:35 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:35.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:35 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:35.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:35 smithi173 ceph-mon[57540]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 93 B/s wr, 240 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:35.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:35 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/1303162588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:35.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:35 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/3704733294' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:57:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[64276]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[64276]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[64276]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 93 B/s wr, 240 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/1303162588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:36.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/3704733294' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:57:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[66756]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[66756]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[66756]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 93 B/s wr, 240 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/1303162588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:36.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:35 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/3704733294' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:57:37.003 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:36 smithi092 ceph-mon[64276]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:37.003 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:36 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:37.003 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:36 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:37.003 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:36 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:37.003 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:36 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:37.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:36 smithi092 ceph-mon[66756]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:37.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:36 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:37.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:36 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:37.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:36 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:37.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:36 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:36 smithi173 ceph-mon[57540]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:57:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:36 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:36 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:36 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi173.hvgnvt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:37.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:36 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:37.824 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:37 smithi173 ceph-mon[57540]: Upgrade: Updating rgw.foo.smithi173.hvgnvt (3/4) 2024-06-21T01:57:37.824 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:37 smithi173 ceph-mon[57540]: Deploying daemon rgw.foo.smithi173.hvgnvt on smithi173 2024-06-21T01:57:37.824 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:37 smithi173 ceph-mon[57540]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 170 B/s wr, 266 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:38.014 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:37 smithi092 ceph-mon[64276]: Upgrade: Updating rgw.foo.smithi173.hvgnvt (3/4) 2024-06-21T01:57:38.014 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:37 smithi092 ceph-mon[64276]: Deploying daemon rgw.foo.smithi173.hvgnvt on smithi173 2024-06-21T01:57:38.014 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:37 smithi092 ceph-mon[64276]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 170 B/s wr, 266 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:38.015 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:37 smithi092 ceph-mon[66756]: Upgrade: Updating rgw.foo.smithi173.hvgnvt (3/4) 2024-06-21T01:57:38.015 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:37 smithi092 ceph-mon[66756]: Deploying daemon rgw.foo.smithi173.hvgnvt on smithi173 2024-06-21T01:57:38.015 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:37 smithi092 ceph-mon[66756]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s rd, 170 B/s wr, 266 op/s; 0 B/s, 0 objects/s recovering 2024-06-21T01:57:39.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:39 smithi173 ceph-mon[57540]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 170 B/s wr, 315 op/s 2024-06-21T01:57:39.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:39 smithi092 ceph-mon[64276]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 170 B/s wr, 315 op/s 2024-06-21T01:57:39.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:39 smithi092 ceph-mon[66756]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 170 B/s wr, 315 op/s 2024-06-21T01:57:39.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:57:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:57:39] "GET /metrics HTTP/1.1" 200 38616 "" "Prometheus/2.33.4" 2024-06-21T01:57:41.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:40 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:41.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:40 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:41.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:40 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:41.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:40 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:41.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:40 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:41.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:40 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:41 smithi173 ceph-mon[57540]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 180 KiB/s rd, 170 B/s wr, 285 op/s 2024-06-21T01:57:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:42.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:41 smithi092 ceph-mon[64276]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 180 KiB/s rd, 170 B/s wr, 285 op/s 2024-06-21T01:57:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:42.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:42.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:42.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:41 smithi092 ceph-mon[66756]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 180 KiB/s rd, 170 B/s wr, 285 op/s 2024-06-21T01:57:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi173.khwpnd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-21T01:57:42.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:42.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:42 smithi173 ceph-mon[57540]: Upgrade: Updating rgw.smpl.smithi173.khwpnd (4/4) 2024-06-21T01:57:42.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:42 smithi173 ceph-mon[57540]: Deploying daemon rgw.smpl.smithi173.khwpnd on smithi173 2024-06-21T01:57:43.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:42 smithi092 ceph-mon[66756]: Upgrade: Updating rgw.smpl.smithi173.khwpnd (4/4) 2024-06-21T01:57:43.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:42 smithi092 ceph-mon[66756]: Deploying daemon rgw.smpl.smithi173.khwpnd on smithi173 2024-06-21T01:57:43.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:42 smithi092 ceph-mon[64276]: Upgrade: Updating rgw.smpl.smithi173.khwpnd (4/4) 2024-06-21T01:57:43.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:42 smithi092 ceph-mon[64276]: Deploying daemon rgw.smpl.smithi173.khwpnd on smithi173 2024-06-21T01:57:44.132 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:43 smithi173 ceph-mon[57540]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 255 B/s wr, 292 op/s 2024-06-21T01:57:44.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:43 smithi092 ceph-mon[64276]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 255 B/s wr, 292 op/s 2024-06-21T01:57:44.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:43 smithi092 ceph-mon[66756]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 255 B/s wr, 292 op/s 2024-06-21T01:57:45.792 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:45 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:45.792 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:45 smithi173 ceph-mon[57540]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 170 B/s wr, 291 op/s 2024-06-21T01:57:45.792 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:45 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:45.792 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:45 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:45.793 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:45 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:45.793 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:45 smithi092 ceph-mon[64276]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 170 B/s wr, 291 op/s 2024-06-21T01:57:45.793 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:45 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:45.793 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:45 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:45.794 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:45 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:45.794 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:45 smithi092 ceph-mon[66756]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s rd, 170 B/s wr, 291 op/s 2024-06-21T01:57:45.794 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:45 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:45.794 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:45 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:47.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:47 smithi092 ceph-mon[64276]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 255 B/s wr, 269 op/s 2024-06-21T01:57:47.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:47 smithi092 ceph-mon[66756]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 255 B/s wr, 269 op/s 2024-06-21T01:57:47.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:47 smithi173 ceph-mon[57540]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 255 B/s wr, 269 op/s 2024-06-21T01:57:49.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:57:49] "GET /metrics HTTP/1.1" 200 38612 "" "Prometheus/2.33.4" 2024-06-21T01:57:49.900 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.900 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.900 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[66756]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 170 B/s wr, 299 op/s 2024-06-21T01:57:49.900 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.901 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.901 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.901 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[64276]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 170 B/s wr, 299 op/s 2024-06-21T01:57:49.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:49 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:49 smithi173 ceph-mon[57540]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 170 B/s wr, 299 op/s 2024-06-21T01:57:49.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:49.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:49 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:50.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:50 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:50.846 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:50 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:50.846 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:50 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:50.846 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:50 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:51.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:50 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:51.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:50 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:51.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:51 smithi173 ceph-mon[57540]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-21T01:57:52.006 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:51 smithi092 ceph-mon[64276]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-21T01:57:52.007 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:51 smithi092 ceph-mon[66756]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-21T01:57:53.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: Detected new or changed devices on smithi092 2024-06-21T01:57:53.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 170 B/s wr, 232 op/s 2024-06-21T01:57:53.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:53.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:57:53.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:57:53.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi092.lzbkvh"}]: dispatch 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi092.lzbkvh"}]': finished 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi173.hvgnvt"}]: dispatch 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi173.hvgnvt"}]': finished 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi092.bxxcjp"}]: dispatch 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi092.bxxcjp"}]': finished 2024-06-21T01:57:53.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi173.khwpnd"}]: dispatch 2024-06-21T01:57:53.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi173.khwpnd"}]': finished 2024-06-21T01:57:53.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.716 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: Detected new or changed devices on smithi092 2024-06-21T01:57:53.716 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.716 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.716 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 170 B/s wr, 232 op/s 2024-06-21T01:57:53.716 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.717 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi092.lzbkvh"}]: dispatch 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi092.lzbkvh"}]': finished 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi173.hvgnvt"}]: dispatch 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi173.hvgnvt"}]': finished 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi092.bxxcjp"}]: dispatch 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi092.bxxcjp"}]': finished 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi173.khwpnd"}]: dispatch 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi173.khwpnd"}]': finished 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:53 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: Detected new or changed devices on smithi092 2024-06-21T01:57:53.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 170 B/s wr, 232 op/s 2024-06-21T01:57:53.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi092.lzbkvh"}]: dispatch 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi092.lzbkvh"}]': finished 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi173.hvgnvt"}]: dispatch 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi173.hvgnvt"}]': finished 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi092.bxxcjp"}]: dispatch 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi092.bxxcjp"}]': finished 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi173.khwpnd"}]: dispatch 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi173.khwpnd"}]': finished 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:57:53.730 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:53 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:54.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:54 smithi173 ceph-mon[57540]: Detected new or changed devices on smithi173 2024-06-21T01:57:54.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:54 smithi173 ceph-mon[57540]: Upgrade: Setting container_image for all rgw 2024-06-21T01:57:54.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:54 smithi173 ceph-mon[57540]: Upgrade: Setting container_image for all rbd-mirror 2024-06-21T01:57:54.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:54 smithi173 ceph-mon[57540]: Upgrade: Setting container_image for all ceph-exporter 2024-06-21T01:57:54.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:54 smithi092 ceph-mon[64276]: Detected new or changed devices on smithi173 2024-06-21T01:57:54.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:54 smithi092 ceph-mon[64276]: Upgrade: Setting container_image for all rgw 2024-06-21T01:57:54.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:54 smithi092 ceph-mon[64276]: Upgrade: Setting container_image for all rbd-mirror 2024-06-21T01:57:54.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:54 smithi092 ceph-mon[64276]: Upgrade: Setting container_image for all ceph-exporter 2024-06-21T01:57:54.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:54 smithi092 ceph-mon[66756]: Detected new or changed devices on smithi173 2024-06-21T01:57:54.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:54 smithi092 ceph-mon[66756]: Upgrade: Setting container_image for all rgw 2024-06-21T01:57:54.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:54 smithi092 ceph-mon[66756]: Upgrade: Setting container_image for all rbd-mirror 2024-06-21T01:57:54.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:54 smithi092 ceph-mon[66756]: Upgrade: Setting container_image for all ceph-exporter 2024-06-21T01:57:55.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:55 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:55.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:55 smithi173 ceph-mon[57540]: Upgrade: Updating iscsi.foo.smithi092.pqnhsu 2024-06-21T01:57:55.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:55 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:55.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:55 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:57:55.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:55 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:55.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:55 smithi173 ceph-mon[57540]: Deploying daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:57:55.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:55 smithi173 ceph-mon[57540]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 85 B/s wr, 187 op/s 2024-06-21T01:57:56.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:56.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[64276]: Upgrade: Updating iscsi.foo.smithi092.pqnhsu 2024-06-21T01:57:56.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:56.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:57:56.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:56.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[64276]: Deploying daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:57:56.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[64276]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 85 B/s wr, 187 op/s 2024-06-21T01:57:56.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:57:56.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[66756]: Upgrade: Updating iscsi.foo.smithi092.pqnhsu 2024-06-21T01:57:56.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:57:56.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.pqnhsu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-21T01:57:56.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:57:56.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[66756]: Deploying daemon iscsi.foo.smithi092.pqnhsu on smithi092 2024-06-21T01:57:56.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:55 smithi092 ceph-mon[66756]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 85 B/s wr, 187 op/s 2024-06-21T01:57:56.784 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:56.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:56.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:57:57.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:57 smithi173 ceph-mon[57540]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 85 B/s wr, 134 op/s 2024-06-21T01:57:58.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:57 smithi092 ceph-mon[64276]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 85 B/s wr, 134 op/s 2024-06-21T01:57:58.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:57 smithi092 ceph-mon[66756]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 85 B/s wr, 134 op/s 2024-06-21T01:57:59.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:57:59 smithi092 ceph-mon[64276]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 103 op/s 2024-06-21T01:57:59.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:57:59 smithi092 ceph-mon[66756]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 103 op/s 2024-06-21T01:57:59.351 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:57:59 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:57:59] "GET /metrics HTTP/1.1" 200 38680 "" "Prometheus/2.33.4" 2024-06-21T01:57:59.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:57:59 smithi173 ceph-mon[57540]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 103 op/s 2024-06-21T01:58:01.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:01 smithi092 ceph-mon[64276]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 26 op/s 2024-06-21T01:58:01.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:01 smithi092 ceph-mon[66756]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 26 op/s 2024-06-21T01:58:01.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:01 smithi173 ceph-mon[57540]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 26 op/s 2024-06-21T01:58:03.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:03 smithi092 ceph-mon[64276]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:03.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:03 smithi092 ceph-mon[66756]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:03.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:03 smithi173 ceph-mon[57540]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:05 smithi092 ceph-mon[64276]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:05.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:05 smithi092 ceph-mon[64276]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:05.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:05 smithi092 ceph-mon[66756]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:05.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:05 smithi092 ceph-mon[66756]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:05.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:05 smithi173 ceph-mon[57540]: from='client.25225 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:05.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:05 smithi173 ceph-mon[57540]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:06.080 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:58:06.727 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (7m) 18s ago 11m 22.0M - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (6m) 17s ago 10m 55.2M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 running (6m) 18s ago 10m 56.3M - 3.5 e1d6a67b021e 3192bc59456b 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (6m) 17s ago 16m 441M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (6m) 18s ago 17m 510M - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (6m) 18s ago 17m 57.2M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (5m) 17s ago 16m 46.2M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (5m) 18s ago 16m 46.5M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (7m) 18s ago 12m 17.0M - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (6m) 17s ago 12m 19.6M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (4m) 18s ago 15m 65.6M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 7c252b0f2dfd 2024-06-21T01:58:06.728 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (4m) 18s ago 15m 59.5M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 9ad8cda61390 2024-06-21T01:58:06.729 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (3m) 18s ago 14m 59.2M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 0fe024e7dad9 2024-06-21T01:58:06.729 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (3m) 18s ago 14m 68.5M 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 f0817e4dfec7 2024-06-21T01:58:06.729 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (2m) 17s ago 14m 58.2M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 d42a6ffd2e66 2024-06-21T01:58:06.729 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (118s) 17s ago 13m 60.5M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 b6050a92252e 2024-06-21T01:58:06.729 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (86s) 17s ago 13m 59.9M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 dffac360ac20 2024-06-21T01:58:06.729 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (52s) 17s ago 13m 62.4M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 3d835c4a13e6 2024-06-21T01:58:06.729 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (6m) 17s ago 12m 75.1M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:58:06.729 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (38s) 18s ago 10m 75.9M - 19.0.0-2541-gc65be810 e9bfc9e474e5 81182104b6c5 2024-06-21T01:58:06.729 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (27s) 17s ago 10m 76.5M - 19.0.0-2541-gc65be810 e9bfc9e474e5 c15966b0ec45 2024-06-21T01:58:06.729 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (32s) 18s ago 10m 76.7M - 19.0.0-2541-gc65be810 e9bfc9e474e5 24e66f987cfe 2024-06-21T01:58:06.729 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (22s) 17s ago 10m 76.6M - 19.0.0-2541-gc65be810 e9bfc9e474e5 92f26aa364c2 2024-06-21T01:58:07.136 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:58:07.136 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 8 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 4 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 17 2024-06-21T01:58:07.137 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:58:07.138 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:58:07.333 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:07 smithi092 ceph-mon[64276]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:07.333 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:07 smithi092 ceph-mon[64276]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:07.333 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:07 smithi092 ceph-mon[64276]: from='client.44584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:07.333 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:07 smithi092 ceph-mon[64276]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:07.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:07 smithi092 ceph-mon[66756]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:07.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:07 smithi092 ceph-mon[66756]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:07.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:07 smithi092 ceph-mon[66756]: from='client.44584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:07.334 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:07 smithi092 ceph-mon[66756]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:07.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:07 smithi173 ceph-mon[57540]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:07.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:07 smithi173 ceph-mon[57540]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:07.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:07 smithi173 ceph-mon[57540]: from='client.44584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:07.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:07 smithi173 ceph-mon[57540]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:07.487 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:58:07.487 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:58:07.487 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:58:07.487 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:58:07.487 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:58:07.487 INFO:teuthology.orchestra.run.smithi092.stdout: "osd", 2024-06-21T01:58:07.487 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr", 2024-06-21T01:58:07.488 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw", 2024-06-21T01:58:07.488 INFO:teuthology.orchestra.run.smithi092.stdout: "mon" 2024-06-21T01:58:07.488 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:58:07.488 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "17/23 daemons upgraded", 2024-06-21T01:58:07.488 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Currently upgrading iscsi daemons", 2024-06-21T01:58:07.488 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": false 2024-06-21T01:58:07.488 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:58:07.921 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_OK 2024-06-21T01:58:08.272 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:08 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/2749359138' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:58:08.272 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:08 smithi092 ceph-mon[64276]: from='client.44596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:08.272 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:08 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/1561531687' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:58:08.272 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:08 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/2749359138' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:58:08.272 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:08 smithi092 ceph-mon[66756]: from='client.44596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:08.272 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:08 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/1561531687' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:58:08.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:08 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/2749359138' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:58:08.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:08 smithi173 ceph-mon[57540]: from='client.44596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:08.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:08 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/1561531687' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:58:08.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:58:08 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T01:58:08.310+0000 7f995778a640 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T01:58:09.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:09 smithi092 ceph-mon[64276]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T01:58:09.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:09 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:09.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:09 smithi092 ceph-mon[64276]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-21T01:58:09.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:09 smithi092 ceph-mon[66756]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T01:58:09.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:09 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:09.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:09 smithi092 ceph-mon[66756]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-21T01:58:09.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:58:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:58:09] "GET /metrics HTTP/1.1" 200 38680 "" "Prometheus/2.33.4" 2024-06-21T01:58:09.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:09 smithi173 ceph-mon[57540]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T01:58:09.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:09 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:09.728 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:09 smithi173 ceph-mon[57540]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-21T01:58:10.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:10 smithi092 ceph-mon[64276]: Health check failed: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-06-21T01:58:10.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:10 smithi092 ceph-mon[66756]: Health check failed: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-06-21T01:58:10.598 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:10 smithi173 ceph-mon[57540]: Health check failed: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-06-21T01:58:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:11 smithi173 ceph-mon[57540]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-06-21T01:58:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:11 smithi092 ceph-mon[64276]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-06-21T01:58:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:12.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:11 smithi092 ceph-mon[66756]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-06-21T01:58:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:12.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:58:13.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:58:13.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-06-21T01:58:13.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: Checking dashboard <-> RGW credentials 2024-06-21T01:58:13.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:58:13.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:58:13.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 533 B/s rd, 0 op/s 2024-06-21T01:58:13.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 673 B/s rd, 0 op/s 2024-06-21T01:58:13.479 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:58:13.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:58:13.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-06-21T01:58:13.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: Checking dashboard <-> RGW credentials 2024-06-21T01:58:13.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:58:13.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:58:13.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 533 B/s rd, 0 op/s 2024-06-21T01:58:13.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 673 B/s rd, 0 op/s 2024-06-21T01:58:13.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:58:13.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:58:13.602 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-06-21T01:58:13.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: Checking dashboard <-> RGW credentials 2024-06-21T01:58:13.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:13.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:58:13.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:58:13.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 533 B/s rd, 0 op/s 2024-06-21T01:58:13.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 673 B/s rd, 0 op/s 2024-06-21T01:58:13.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:14.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:14 smithi173 ceph-mon[57540]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-21T01:58:14.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:14 smithi092 ceph-mon[64276]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-21T01:58:14.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:14 smithi092 ceph-mon[66756]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-21T01:58:15.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:15 smithi173 ceph-mon[57540]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-06-21T01:58:15.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:15 smithi092 ceph-mon[64276]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-06-21T01:58:15.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:15 smithi092 ceph-mon[66756]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-06-21T01:58:17.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:17 smithi173 ceph-mon[57540]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-06-21T01:58:18.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:17 smithi092 ceph-mon[64276]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-06-21T01:58:18.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:17 smithi092 ceph-mon[66756]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-06-21T01:58:19.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:58:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:58:19] "GET /metrics HTTP/1.1" 200 38761 "" "Prometheus/2.33.4" 2024-06-21T01:58:19.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:19 smithi173 ceph-mon[57540]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-06-21T01:58:19.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:19 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/984792217' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:58:20.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:19 smithi092 ceph-mon[64276]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-06-21T01:58:20.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:19 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/984792217' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:58:20.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:19 smithi092 ceph-mon[66756]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-06-21T01:58:20.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:19 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/984792217' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-21T01:58:20.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:20 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/133577985' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/914442215"}]: dispatch 2024-06-21T01:58:20.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:20 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/133577985' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/914442215"}]: dispatch 2024-06-21T01:58:20.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:20 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/133577985' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/914442215"}]: dispatch 2024-06-21T01:58:21.903 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:21 smithi173 ceph-mon[57540]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-06-21T01:58:21.903 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:21 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/133577985' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/914442215"}]': finished 2024-06-21T01:58:21.903 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:21 smithi173 ceph-mon[57540]: osdmap e151: 8 total, 8 up, 8 in 2024-06-21T01:58:21.903 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:21 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/3117477948' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3236143987"}]: dispatch 2024-06-21T01:58:22.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:21 smithi092 ceph-mon[64276]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-06-21T01:58:22.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:21 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/133577985' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/914442215"}]': finished 2024-06-21T01:58:22.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:21 smithi092 ceph-mon[64276]: osdmap e151: 8 total, 8 up, 8 in 2024-06-21T01:58:22.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:21 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/3117477948' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3236143987"}]: dispatch 2024-06-21T01:58:22.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:21 smithi092 ceph-mon[66756]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-06-21T01:58:22.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:21 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/133577985' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/914442215"}]': finished 2024-06-21T01:58:22.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:21 smithi092 ceph-mon[66756]: osdmap e151: 8 total, 8 up, 8 in 2024-06-21T01:58:22.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:21 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/3117477948' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3236143987"}]: dispatch 2024-06-21T01:58:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:22 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/3117477948' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3236143987"}]': finished 2024-06-21T01:58:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:22 smithi173 ceph-mon[57540]: osdmap e152: 8 total, 8 up, 8 in 2024-06-21T01:58:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:22 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/1197893721' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/397612098"}]: dispatch 2024-06-21T01:58:23.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:22 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/3117477948' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3236143987"}]': finished 2024-06-21T01:58:23.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:22 smithi092 ceph-mon[64276]: osdmap e152: 8 total, 8 up, 8 in 2024-06-21T01:58:23.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:22 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/1197893721' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/397612098"}]: dispatch 2024-06-21T01:58:23.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:22 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/3117477948' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3236143987"}]': finished 2024-06-21T01:58:23.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:22 smithi092 ceph-mon[66756]: osdmap e152: 8 total, 8 up, 8 in 2024-06-21T01:58:23.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:22 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/1197893721' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/397612098"}]: dispatch 2024-06-21T01:58:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:23 smithi173 ceph-mon[57540]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-06-21T01:58:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:23 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/1197893721' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/397612098"}]': finished 2024-06-21T01:58:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:23 smithi173 ceph-mon[57540]: osdmap e153: 8 total, 8 up, 8 in 2024-06-21T01:58:23.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:23 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/2650193612' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2039410236"}]: dispatch 2024-06-21T01:58:24.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:23 smithi092 ceph-mon[64276]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-06-21T01:58:24.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:23 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/1197893721' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/397612098"}]': finished 2024-06-21T01:58:24.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:23 smithi092 ceph-mon[64276]: osdmap e153: 8 total, 8 up, 8 in 2024-06-21T01:58:24.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:23 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/2650193612' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2039410236"}]: dispatch 2024-06-21T01:58:24.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:23 smithi092 ceph-mon[66756]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-06-21T01:58:24.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:23 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/1197893721' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/397612098"}]': finished 2024-06-21T01:58:24.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:23 smithi092 ceph-mon[66756]: osdmap e153: 8 total, 8 up, 8 in 2024-06-21T01:58:24.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:23 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/2650193612' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2039410236"}]: dispatch 2024-06-21T01:58:24.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:24 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/2650193612' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2039410236"}]': finished 2024-06-21T01:58:24.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:24 smithi173 ceph-mon[57540]: osdmap e154: 8 total, 8 up, 8 in 2024-06-21T01:58:24.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:24 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/3196227925' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1765960496"}]: dispatch 2024-06-21T01:58:24.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:24 smithi173 ceph-mon[57540]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1765960496"}]: dispatch 2024-06-21T01:58:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:24 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/2650193612' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2039410236"}]': finished 2024-06-21T01:58:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:24 smithi092 ceph-mon[64276]: osdmap e154: 8 total, 8 up, 8 in 2024-06-21T01:58:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:24 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/3196227925' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1765960496"}]: dispatch 2024-06-21T01:58:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:24 smithi092 ceph-mon[64276]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1765960496"}]: dispatch 2024-06-21T01:58:25.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:24 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/2650193612' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2039410236"}]': finished 2024-06-21T01:58:25.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:24 smithi092 ceph-mon[66756]: osdmap e154: 8 total, 8 up, 8 in 2024-06-21T01:58:25.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:24 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/3196227925' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1765960496"}]: dispatch 2024-06-21T01:58:25.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:24 smithi092 ceph-mon[66756]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1765960496"}]: dispatch 2024-06-21T01:58:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:25 smithi173 ceph-mon[57540]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-06-21T01:58:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:25 smithi173 ceph-mon[57540]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1765960496"}]': finished 2024-06-21T01:58:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:25 smithi173 ceph-mon[57540]: osdmap e155: 8 total, 8 up, 8 in 2024-06-21T01:58:25.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:25 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/2034079799' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/397612098"}]: dispatch 2024-06-21T01:58:25.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:25 smithi173 ceph-mon[57540]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/397612098"}]: dispatch 2024-06-21T01:58:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:25 smithi092 ceph-mon[64276]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-06-21T01:58:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:25 smithi092 ceph-mon[64276]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1765960496"}]': finished 2024-06-21T01:58:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:25 smithi092 ceph-mon[64276]: osdmap e155: 8 total, 8 up, 8 in 2024-06-21T01:58:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:25 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/2034079799' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/397612098"}]: dispatch 2024-06-21T01:58:26.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:25 smithi092 ceph-mon[64276]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/397612098"}]: dispatch 2024-06-21T01:58:26.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:25 smithi092 ceph-mon[66756]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-06-21T01:58:26.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:25 smithi092 ceph-mon[66756]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1765960496"}]': finished 2024-06-21T01:58:26.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:25 smithi092 ceph-mon[66756]: osdmap e155: 8 total, 8 up, 8 in 2024-06-21T01:58:26.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:25 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/2034079799' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/397612098"}]: dispatch 2024-06-21T01:58:26.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:25 smithi092 ceph-mon[66756]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/397612098"}]: dispatch 2024-06-21T01:58:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:26 smithi173 ceph-mon[57540]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/397612098"}]': finished 2024-06-21T01:58:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:26 smithi173 ceph-mon[57540]: osdmap e156: 8 total, 8 up, 8 in 2024-06-21T01:58:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:27.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:26 smithi092 ceph-mon[64276]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/397612098"}]': finished 2024-06-21T01:58:27.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:26 smithi092 ceph-mon[64276]: osdmap e156: 8 total, 8 up, 8 in 2024-06-21T01:58:27.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:27.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:27.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:26 smithi092 ceph-mon[66756]: from='client.? ' entity='client.iscsi.foo.smithi092.pqnhsu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/397612098"}]': finished 2024-06-21T01:58:27.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:26 smithi092 ceph-mon[66756]: osdmap e156: 8 total, 8 up, 8 in 2024-06-21T01:58:27.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:58:27.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:28.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:27 smithi173 ceph-mon[57540]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:58:28.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:27 smithi092 ceph-mon[64276]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:58:28.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:27 smithi092 ceph-mon[66756]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail 2024-06-21T01:58:29.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:58:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:58:29] "GET /metrics HTTP/1.1" 200 38759 "" "Prometheus/2.33.4" 2024-06-21T01:58:30.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:29 smithi173 ceph-mon[57540]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-06-21T01:58:30.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:29 smithi092 ceph-mon[66756]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-06-21T01:58:30.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:29 smithi092 ceph-mon[64276]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-06-21T01:58:31.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:30 smithi173 ceph-mon[57540]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:31.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:30 smithi092 ceph-mon[64276]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:31.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:30 smithi092 ceph-mon[66756]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:32.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:31 smithi173 ceph-mon[57540]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-06-21T01:58:32.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:31 smithi092 ceph-mon[64276]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-06-21T01:58:32.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:31 smithi092 ceph-mon[66756]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-06-21T01:58:34.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:33 smithi173 ceph-mon[57540]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:58:34.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:33 smithi092 ceph-mon[64276]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:58:34.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:33 smithi092 ceph-mon[66756]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-21T01:58:36.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:35 smithi173 ceph-mon[57540]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-06-21T01:58:36.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:35 smithi092 ceph-mon[64276]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-06-21T01:58:36.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:35 smithi092 ceph-mon[66756]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-06-21T01:58:38.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:37 smithi173 ceph-mon[57540]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-06-21T01:58:38.281 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:37 smithi092 ceph-mon[64276]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-06-21T01:58:38.281 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:37 smithi092 ceph-mon[66756]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-06-21T01:58:38.281 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T01:58:38.608 INFO:teuthology.orchestra.run.smithi092.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed." 2024-06-21T01:58:39.343 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph orch ps' 2024-06-21T01:58:39.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:58:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:58:39] "GET /metrics HTTP/1.1" 200 38759 "" "Prometheus/2.33.4" 2024-06-21T01:58:40.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:39 smithi173 ceph-mon[57540]: from='client.34779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:40.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:39 smithi173 ceph-mon[57540]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:40.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:39 smithi173 ceph-mon[57540]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:40.339 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:39 smithi092 ceph-mon[64276]: from='client.34779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:40.339 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:39 smithi092 ceph-mon[64276]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:40.339 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:39 smithi092 ceph-mon[64276]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:40.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:39 smithi092 ceph-mon[66756]: from='client.34779 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:40.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:39 smithi092 ceph-mon[66756]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:40.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:39 smithi092 ceph-mon[66756]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:41.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:40 smithi173 ceph-mon[57540]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:41.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:40 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:41.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:40 smithi092 ceph-mon[64276]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:41.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:40 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:41.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:40 smithi092 ceph-mon[66756]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:41.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:40 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:41.385 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:58:41.385 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (7m) 29s ago 12m - - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:58:41.385 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (7m) 29s ago 11m 55.2M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:58:41.385 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 unknown 29s ago 11m - - 2024-06-21T01:58:41.385 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (6m) 29s ago 16m 441M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:58:41.385 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (7m) 29s ago 18m - - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:58:41.386 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (6m) 29s ago 18m - 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:58:41.386 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (6m) 29s ago 16m 44.8M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:58:41.386 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (6m) 29s ago 17m - 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:58:41.386 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (7m) 29s ago 12m - - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:58:41.386 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (7m) 29s ago 12m 17.9M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:58:41.386 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (5m) 29s ago 16m - 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 7c252b0f2dfd 2024-06-21T01:58:41.386 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (4m) 29s ago 15m - 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 9ad8cda61390 2024-06-21T01:58:41.386 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (4m) 29s ago 15m - 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 0fe024e7dad9 2024-06-21T01:58:41.386 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (3m) 29s ago 15m - 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 f0817e4dfec7 2024-06-21T01:58:41.386 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (3m) 29s ago 14m 58.4M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 d42a6ffd2e66 2024-06-21T01:58:41.387 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (2m) 29s ago 14m 60.7M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 b6050a92252e 2024-06-21T01:58:41.387 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (2m) 29s ago 14m 60.1M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 dffac360ac20 2024-06-21T01:58:41.387 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (86s) 29s ago 13m 61.5M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 3d835c4a13e6 2024-06-21T01:58:41.387 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (6m) 29s ago 13m 75.1M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:58:41.387 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (72s) 29s ago 11m - - 19.0.0-2541-gc65be810 e9bfc9e474e5 81182104b6c5 2024-06-21T01:58:41.387 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (61s) 29s ago 11m 76.8M - 19.0.0-2541-gc65be810 e9bfc9e474e5 c15966b0ec45 2024-06-21T01:58:41.387 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (67s) 29s ago 11m - - 19.0.0-2541-gc65be810 e9bfc9e474e5 24e66f987cfe 2024-06-21T01:58:41.387 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (56s) 29s ago 11m 76.9M - 19.0.0-2541-gc65be810 e9bfc9e474e5 92f26aa364c2 2024-06-21T01:58:41.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph versions' 2024-06-21T01:58:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:41 smithi173 ceph-mon[57540]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:58:42.282 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:41 smithi092 ceph-mon[64276]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:58:42.283 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:41 smithi092 ceph-mon[66756]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-21T01:58:43.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:42 smithi173 ceph-mon[57540]: from='client.34791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:43.244 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:42 smithi092 ceph-mon[64276]: from='client.34791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:43.244 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:42 smithi092 ceph-mon[66756]: from='client.34791 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:58:43.859 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 8 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 4 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:58:43.860 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:58:43.861 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 17 2024-06-21T01:58:43.861 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:58:43.861 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:58:44.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:43 smithi173 ceph-mon[57540]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:44.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:43 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/4208591005' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:58:44.324 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:43 smithi092 ceph-mon[64276]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:44.324 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:43 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/4208591005' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:58:44.324 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:43 smithi092 ceph-mon[66756]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:44.324 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:43 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/4208591005' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:58:44.465 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-06-21T01:58:46.143 INFO:teuthology.orchestra.run.smithi092.stdout:wait for servicemap items w/ changing names to refresh 2024-06-21T01:58:46.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:45 smithi173 ceph-mon[57540]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:46.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:45 smithi092 ceph-mon[64276]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:46.351 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:45 smithi092 ceph-mon[66756]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:46.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'sleep 60' 2024-06-21T01:58:47.283 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:47 smithi092 ceph-mon[64276]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:47.283 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:47 smithi092 ceph-mon[66756]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:47.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:47 smithi173 ceph-mon[57540]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:49.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:58:49 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:58:49] "GET /metrics HTTP/1.1" 200 38756 "" "Prometheus/2.33.4" 2024-06-21T01:58:49.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:49 smithi173 ceph-mon[57540]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:50.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:49 smithi092 ceph-mon[64276]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:50.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:49 smithi092 ceph-mon[66756]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:50.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:50 smithi173 ceph-mon[57540]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:51.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:50 smithi092 ceph-mon[64276]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:51.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:50 smithi092 ceph-mon[66756]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:58:51.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:51 smithi092 ceph-mon[64276]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:51.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:51 smithi092 ceph-mon[66756]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:51.897 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:51 smithi173 ceph-mon[57540]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:53.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:53 smithi173 ceph-mon[57540]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:54.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:53 smithi092 ceph-mon[64276]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:54.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:53 smithi092 ceph-mon[66756]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:58:55.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:55 smithi173 ceph-mon[57540]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:56.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:55 smithi092 ceph-mon[64276]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:56.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:55 smithi092 ceph-mon[66756]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:58:56.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:57.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:57.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:58:57.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:57 smithi173 ceph-mon[57540]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:58:58.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:57 smithi092 ceph-mon[64276]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:58:58.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:57 smithi092 ceph-mon[66756]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:58:59.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:58:59 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:58:59] "GET /metrics HTTP/1.1" 200 38756 "" "Prometheus/2.33.4" 2024-06-21T01:58:59.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:58:59 smithi173 ceph-mon[57540]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:00.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:58:59 smithi092 ceph-mon[64276]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:00.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:58:59 smithi092 ceph-mon[66756]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:00.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:00 smithi173 ceph-mon[57540]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:01.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:00 smithi092 ceph-mon[64276]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:01.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:00 smithi092 ceph-mon[66756]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:01.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:01 smithi173 ceph-mon[57540]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:02.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:01 smithi092 ceph-mon[64276]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:02.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:01 smithi092 ceph-mon[66756]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:03.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:03 smithi173 ceph-mon[57540]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:04.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:03 smithi092 ceph-mon[64276]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:03 smithi092 ceph-mon[66756]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:05.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:05 smithi173 ceph-mon[57540]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:06.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:05 smithi092 ceph-mon[64276]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:06.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:05 smithi092 ceph-mon[66756]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:07.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:07 smithi173 ceph-mon[57540]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:08.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:07 smithi092 ceph-mon[64276]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:08.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:07 smithi092 ceph-mon[66756]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:09.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:59:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:59:09] "GET /metrics HTTP/1.1" 200 38756 "" "Prometheus/2.33.4" 2024-06-21T01:59:09.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:09 smithi173 ceph-mon[57540]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:10.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:09 smithi092 ceph-mon[64276]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:10.023 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:09 smithi092 ceph-mon[66756]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:10.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:10 smithi173 ceph-mon[57540]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:11.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:10 smithi092 ceph-mon[64276]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:11.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:10 smithi092 ceph-mon[66756]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:11.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:11 smithi173 ceph-mon[57540]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:11.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:12.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:11 smithi092 ceph-mon[64276]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:12.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:12.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:11 smithi092 ceph-mon[66756]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:12.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:13.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:13 smithi173 ceph-mon[57540]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:13.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:59:13.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:59:13.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:59:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:13 smithi092 ceph-mon[64276]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:59:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:59:14.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:59:14.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:13 smithi092 ceph-mon[66756]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:14.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T01:59:14.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T01:59:14.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T01:59:14.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:14 smithi173 ceph-mon[57540]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-06-21T01:59:14.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:14 smithi173 ceph-mon[57540]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:15.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:14 smithi092 ceph-mon[64276]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-06-21T01:59:15.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:14 smithi092 ceph-mon[64276]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:15.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:14 smithi092 ceph-mon[66756]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-06-21T01:59:15.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:14 smithi092 ceph-mon[66756]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:16.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:16 smithi173 ceph-mon[57540]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:17.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:16 smithi092 ceph-mon[64276]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:17.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:16 smithi092 ceph-mon[66756]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:18.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:18 smithi173 ceph-mon[57540]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-06-21T01:59:19.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:18 smithi092 ceph-mon[64276]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-06-21T01:59:19.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:18 smithi092 ceph-mon[66756]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-06-21T01:59:19.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:59:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:59:19] "GET /metrics HTTP/1.1" 200 38754 "" "Prometheus/2.33.4" 2024-06-21T01:59:20.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:20 smithi173 ceph-mon[57540]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:20.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:20 smithi173 ceph-mon[57540]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:21.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:20 smithi092 ceph-mon[64276]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:21.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:20 smithi092 ceph-mon[64276]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:21.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:20 smithi092 ceph-mon[66756]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:21.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:20 smithi092 ceph-mon[66756]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:22.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:22 smithi173 ceph-mon[57540]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-06-21T01:59:23.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:22 smithi092 ceph-mon[64276]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-06-21T01:59:23.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:22 smithi092 ceph-mon[66756]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-06-21T01:59:24.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:24 smithi173 ceph-mon[57540]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:59:25.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:24 smithi092 ceph-mon[64276]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:59:25.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:24 smithi092 ceph-mon[66756]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-21T01:59:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:26 smithi173 ceph-mon[57540]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:26.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:26 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:27.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:26 smithi092 ceph-mon[64276]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:27.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:26 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:27.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:26 smithi092 ceph-mon[66756]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:27.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:26 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:28.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:28 smithi173 ceph-mon[57540]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:29.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:28 smithi092 ceph-mon[64276]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:29.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:28 smithi092 ceph-mon[66756]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:29.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:59:29 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:59:29] "GET /metrics HTTP/1.1" 200 38755 "" "Prometheus/2.33.4" 2024-06-21T01:59:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:30 smithi173 ceph-mon[57540]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:30.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:30 smithi173 ceph-mon[57540]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:30 smithi092 ceph-mon[64276]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:31.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:30 smithi092 ceph-mon[64276]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:31.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:30 smithi092 ceph-mon[66756]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:31.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:30 smithi092 ceph-mon[66756]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:32.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:32 smithi173 ceph-mon[57540]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:33.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:32 smithi092 ceph-mon[66756]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:33.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:32 smithi092 ceph-mon[64276]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:34.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:34 smithi173 ceph-mon[57540]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:35.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:34 smithi092 ceph-mon[64276]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:35.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:34 smithi092 ceph-mon[66756]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:36.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:36 smithi173 ceph-mon[57540]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:37.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:36 smithi092 ceph-mon[64276]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:37.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:36 smithi092 ceph-mon[66756]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-21T01:59:39.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:38 smithi092 ceph-mon[64276]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:39.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:38 smithi092 ceph-mon[66756]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:39.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:38 smithi173 ceph-mon[57540]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:39.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:59:39 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:59:39] "GET /metrics HTTP/1.1" 200 38755 "" "Prometheus/2.33.4" 2024-06-21T01:59:41.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:40 smithi092 ceph-mon[64276]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:41.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:40 smithi092 ceph-mon[64276]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:41.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:40 smithi092 ceph-mon[66756]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:41.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:40 smithi092 ceph-mon[66756]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:41.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:40 smithi173 ceph-mon[57540]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:41.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:40 smithi173 ceph-mon[57540]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:42.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:41 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:42.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:41 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:42.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:41 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:43.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:42 smithi092 ceph-mon[64276]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:43.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:42 smithi092 ceph-mon[66756]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:43.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:42 smithi173 ceph-mon[57540]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:45.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:44 smithi092 ceph-mon[64276]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:45.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:44 smithi092 ceph-mon[66756]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:45.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:44 smithi173 ceph-mon[57540]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:47.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:46 smithi092 ceph-mon[64276]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:47.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:46 smithi092 ceph-mon[66756]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:47.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:46 smithi173 ceph-mon[57540]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:48.992 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:48 smithi092 ceph-mon[64276]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:48.993 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:48 smithi092 ceph-mon[66756]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:49.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph orch ps' 2024-06-21T01:59:49.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:48 smithi173 ceph-mon[57540]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:49.571 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:59:49 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:59:49] "GET /metrics HTTP/1.1" 200 38753 "" "Prometheus/2.33.4" 2024-06-21T01:59:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:50 smithi092 ceph-mon[64276]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:50.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:50 smithi092 ceph-mon[64276]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:50 smithi092 ceph-mon[66756]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:50.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:50 smithi092 ceph-mon[66756]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:51.006 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-21T01:59:51.007 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (8m) 98s ago 13m - - 0.23.0 ba2b418f427c 9b5301758256 2024-06-21T01:59:51.007 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi173 *:3000 running (8m) 99s ago 12m 55.2M - 8.3.5 dad864ee21e9 0003e9fdc5d3 2024-06-21T01:59:51.007 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.pqnhsu smithi092 unknown 98s ago 12m - - 2024-06-21T01:59:51.007 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi173 *:8443,9283,8765 running (8m) 99s ago 17m 441M - 19.0.0-2541-gc65be810 e9bfc9e474e5 6e42ae7481a0 2024-06-21T01:59:51.007 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:8443,9283,8765 running (8m) 98s ago 19m - - 19.0.0-2541-gc65be810 e9bfc9e474e5 fd1d3ac2265b 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (7m) 98s ago 19m - 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 347e7d52844c 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi173 running (7m) 99s ago 18m 44.8M 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 e6b535ec7a55 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (7m) 98s ago 18m - 2048M 19.0.0-2541-gc65be810 e9bfc9e474e5 6549ab7d4df8 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (8m) 98s ago 14m - - 1.3.1 1dbe0e931976 099deaf45853 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi173 *:9100 running (8m) 99s ago 14m 17.9M - 1.3.1 1dbe0e931976 e0cf70cfc752 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (6m) 98s ago 17m - 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 7c252b0f2dfd 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (5m) 98s ago 17m - 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 9ad8cda61390 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (5m) 98s ago 16m - 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 0fe024e7dad9 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (4m) 98s ago 16m - 2736M 19.0.0-2541-gc65be810 e9bfc9e474e5 f0817e4dfec7 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi173 running (4m) 99s ago 15m 58.4M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 d42a6ffd2e66 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi173 running (3m) 99s ago 15m 60.7M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 b6050a92252e 2024-06-21T01:59:51.008 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi173 running (3m) 99s ago 15m 60.1M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 dffac360ac20 2024-06-21T01:59:51.009 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi173 running (2m) 99s ago 14m 61.5M 2992M 19.0.0-2541-gc65be810 e9bfc9e474e5 3d835c4a13e6 2024-06-21T01:59:51.009 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi173 *:9095 running (8m) 99s ago 14m 75.1M - 2.33.4 514e6a882f6e 8e7ba220b26d 2024-06-21T01:59:51.009 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.lzbkvh smithi092 *:8000 running (2m) 98s ago 12m - - 19.0.0-2541-gc65be810 e9bfc9e474e5 81182104b6c5 2024-06-21T01:59:51.009 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi173.hvgnvt smithi173 *:8000 running (2m) 99s ago 12m 76.8M - 19.0.0-2541-gc65be810 e9bfc9e474e5 c15966b0ec45 2024-06-21T01:59:51.009 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.bxxcjp smithi092 *:80 running (2m) 98s ago 12m - - 19.0.0-2541-gc65be810 e9bfc9e474e5 24e66f987cfe 2024-06-21T01:59:51.009 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi173.khwpnd smithi173 *:80 running (2m) 99s ago 12m 76.9M - 19.0.0-2541-gc65be810 e9bfc9e474e5 92f26aa364c2 2024-06-21T01:59:51.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:50 smithi173 ceph-mon[57540]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:51.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:50 smithi173 ceph-mon[57540]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T01:59:51.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph versions' 2024-06-21T01:59:53.030 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:52 smithi173 ceph-mon[57540]: from='client.44728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:59:53.030 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:52 smithi173 ceph-mon[57540]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:53.033 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:52 smithi092 ceph-mon[64276]: from='client.44728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:59:53.033 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:52 smithi092 ceph-mon[64276]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:53.033 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:52 smithi092 ceph-mon[66756]: from='client.44728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:59:53.034 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:52 smithi092 ceph-mon[66756]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:53.683 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:59:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-06-21T01:59:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 3 2024-06-21T01:59:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:59:53.683 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-06-21T01:59:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 2 2024-06-21T01:59:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:59:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-06-21T01:59:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 8 2024-06-21T01:59:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:59:53.684 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-06-21T01:59:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 4 2024-06-21T01:59:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-06-21T01:59:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-06-21T01:59:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)": 17 2024-06-21T01:59:53.685 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-06-21T01:59:53.686 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:59:54.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:53 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/3451983752' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:59:54.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:53 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/3451983752' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:59:54.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:53 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/3451983752' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T01:59:54.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph orch upgrade status' 2024-06-21T01:59:54.997 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:54 smithi092 ceph-mon[64276]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:54.998 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:54 smithi092 ceph-mon[66756]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:55.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:54 smithi173 ceph-mon[57540]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T01:59:56.541 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd", 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [ 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "osd", 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr", 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw", 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "mon" 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "17/23 daemons upgraded", 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed.", 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout: "is_paused": true 2024-06-21T01:59:56.542 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T01:59:56.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:56 smithi092 ceph-mon[66756]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:56.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:56 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:56.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:56 smithi092 ceph-mon[64276]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:56.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:56 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:57.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph health detail' 2024-06-21T01:59:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:56 smithi173 ceph-mon[57540]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:57.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:56 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T01:59:58.065 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:57 smithi092 ceph-mon[64276]: from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:59:58.065 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:57 smithi092 ceph-mon[66756]: from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:59:58.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:57 smithi173 ceph-mon[57540]: from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T01:59:59.100 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:58 smithi092 ceph-mon[64276]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:59.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:58 smithi092 ceph-mon[66756]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:59.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:58 smithi173 ceph-mon[57540]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T01:59:59.237 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T01:59:59.237 INFO:teuthology.orchestra.run.smithi092.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-21T01:59:59.237 INFO:teuthology.orchestra.run.smithi092.stdout: daemon iscsi.foo.smithi092.pqnhsu on smithi092 is in unknown state 2024-06-21T01:59:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T01:59:59.238 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrade daemon: iscsi.foo.smithi092.pqnhsu: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T01:59:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Creating ceph-iscsi config... 2024-06-21T01:59:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T01:59:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T01:59:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d22eef780\x2d2f6f\x2d11ef\x2dbc9d\x2dc7b262605968.slice/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service 2024-06-21T01:59:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 1 from systemctl start ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T01:59:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:59:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:59:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Traceback (most recent call last): 2024-06-21T01:59:59.238 INFO:teuthology.orchestra.run.smithi092.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T01:59:59.239 INFO:teuthology.orchestra.run.smithi092.stdout: return _run_code(code, main_globals, None, 2024-06-21T01:59:59.239 INFO:teuthology.orchestra.run.smithi092.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T01:59:59.239 INFO:teuthology.orchestra.run.smithi092.stdout: exec(code, run_globals) 2024-06-21T01:59:59.239 INFO:teuthology.orchestra.run.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T01:59:59.239 INFO:teuthology.orchestra.run.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T01:59:59.239 INFO:teuthology.orchestra.run.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T01:59:59.239 INFO:teuthology.orchestra.run.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T01:59:59.239 INFO:teuthology.orchestra.run.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T01:59:59.239 INFO:teuthology.orchestra.run.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1087, in deploy_daemon 2024-06-21T01:59:59.240 INFO:teuthology.orchestra.run.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1223, in deploy_daemon_units 2024-06-21T01:59:59.240 INFO:teuthology.orchestra.run.smithi092.stdout: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T01:59:59.240 INFO:teuthology.orchestra.run.smithi092.stdout:RuntimeError: Failed command: systemctl start ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T01:59:59.240 INFO:teuthology.orchestra.run.smithi092.stdout:See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T01:59:59.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 01:59:59 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:01:59:59] "GET /metrics HTTP/1.1" 200 38758 "" "Prometheus/2.33.4" 2024-06-21T01:59:59.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 01:59:59 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/4171245946' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:59:59.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 01:59:59 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/4171245946' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T01:59:59.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-06-21T02:00:00.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 01:59:59 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/4171245946' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-21T02:00:01.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:01.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T02:00:01.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: daemon iscsi.foo.smithi092.pqnhsu on smithi092 is in unknown state 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: Upgrade daemon: iscsi.foo.smithi092.pqnhsu: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: Creating ceph-iscsi config... 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d22eef780\x2d2f6f\x2d11ef\x2dbc9d\x2dc7b262605968.slice/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: Non-zero exit code 1 from systemctl start ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: Traceback (most recent call last): 2024-06-21T02:00:01.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: return _run_code(code, main_globals, None, 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: exec(code, run_globals) 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1087, in deploy_daemon 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1223, in deploy_daemon_units 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: RuntimeError: Failed command: systemctl start ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T02:00:01.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[66756]: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: daemon iscsi.foo.smithi092.pqnhsu on smithi092 is in unknown state 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: Upgrade daemon: iscsi.foo.smithi092.pqnhsu: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: Creating ceph-iscsi config... 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d22eef780\x2d2f6f\x2d11ef\x2dbc9d\x2dc7b262605968.slice/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service 2024-06-21T02:00:01.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: Non-zero exit code 1 from systemctl start ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: Traceback (most recent call last): 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: return _run_code(code, main_globals, None, 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: exec(code, run_globals) 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1087, in deploy_daemon 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1223, in deploy_daemon_units 2024-06-21T02:00:01.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T02:00:01.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: RuntimeError: Failed command: systemctl start ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T02:00:01.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:00 smithi092 ceph-mon[64276]: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T02:00:01.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:01.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: daemon iscsi.foo.smithi092.pqnhsu on smithi092 is in unknown state 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi092.pqnhsu on host smithi092 failed. 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: Upgrade daemon: iscsi.foo.smithi092.pqnhsu: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi092.pqnhsu ... 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: Creating ceph-iscsi config... 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/iscsi-gateway.cfg 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: Write file: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/iscsi.foo.smithi092.pqnhsu/tcmu-runner-entrypoint.sh 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d22eef780\x2d2f6f\x2d11ef\x2dbc9d\x2dc7b262605968.slice/ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: Non-zero exit code 1 from systemctl start ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: systemctl: stderr Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: systemctl: stderr See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: Traceback (most recent call last): 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-21T02:00:01.229 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: return _run_code(code, main_globals, None, 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: exec(code, run_globals) 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5618, in 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 5606, in main 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3096, in command_deploy_from 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3131, in _common_deploy 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 3151, in _deploy_daemon_container 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1087, in deploy_daemon 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: File "/tmp/tmpnc17e2bk.cephadm.build/app/__main__.py", line 1223, in deploy_daemon_units 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: File "/tmp/tmpnc17e2bk.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: RuntimeError: Failed command: systemctl start ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu: Job for ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service failed because the control process exited with error code. 2024-06-21T02:00:01.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:00 smithi173 ceph-mon[57540]: See "systemctl status ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" and "journalctl -xeu ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@iscsi.foo.smithi092.pqnhsu.service" for details. 2024-06-21T02:00:02.107 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-06-21T02:00:02.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-06-21T02:00:03.063 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:02 smithi092 ceph-mon[64276]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:03.063 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:02 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/3986360855' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T02:00:03.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:02 smithi092 ceph-mon[66756]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:03.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:02 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/3986360855' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T02:00:03.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:02 smithi173 ceph-mon[57540]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:03.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:02 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/3986360855' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T02:00:04.835 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 19.0.0-2541-gc65be810 (c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd) squid (rc)" 2024-06-21T02:00:05.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:04 smithi092 ceph-mon[64276]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:05.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:04 smithi092 ceph-mon[66756]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:05.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:04 smithi173 ceph-mon[57540]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:05.448 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -e sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-06-21T02:00:06.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:05 smithi092 ceph-mon[64276]: from='client.? 172.21.15.92:0/3878034423' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T02:00:06.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:05 smithi092 ceph-mon[66756]: from='client.? 172.21.15.92:0/3878034423' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T02:00:06.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:05 smithi173 ceph-mon[57540]: from='client.? 172.21.15.92:0/3878034423' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-21T02:00:06.949 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:06 smithi092 ceph-mon[64276]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:06.950 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:06 smithi092 ceph-mon[66756]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:07.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:06 smithi173 ceph-mon[57540]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:07.373 INFO:teuthology.orchestra.run.smithi092.stdout:osd 8 115s ago - 2024-06-21T02:00:07.994 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-21T02:00:08.003 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi092.front.sepia.ceph.com 2024-06-21T02:00:08.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-06-21T02:00:08.968 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:08 smithi092 ceph-mon[64276]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:08.968 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:08 smithi092 ceph-mon[64276]: from='client.44764 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:08.969 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:08 smithi092 ceph-mon[66756]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:08.969 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:08 smithi092 ceph-mon[66756]: from='client.44764 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:09.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:08 smithi173 ceph-mon[57540]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:09.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:08 smithi173 ceph-mon[57540]: from='client.44764 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:09.501 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:09 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:02:00:09] "GET /metrics HTTP/1.1" 200 38758 "" "Prometheus/2.33.4" 2024-06-21T02:00:11.048 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:10 smithi092 ceph-mon[64276]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:11.048 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:10 smithi092 ceph-mon[64276]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T02:00:11.048 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:10 smithi092 ceph-mon[66756]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:11.048 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:10 smithi092 ceph-mon[66756]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T02:00:11.086 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-06-21T02:00:11.086 INFO:teuthology.orchestra.run.smithi092.stdout: "image": "quay.io/ceph/ceph", 2024-06-21T02:00:11.086 INFO:teuthology.orchestra.run.smithi092.stdout: "registry": "quay.io", 2024-06-21T02:00:11.086 INFO:teuthology.orchestra.run.smithi092.stdout: "bare_image": "ceph/ceph", 2024-06-21T02:00:11.087 INFO:teuthology.orchestra.run.smithi092.stdout: "versions": [] 2024-06-21T02:00:11.087 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-06-21T02:00:11.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:10 smithi173 ceph-mon[57540]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:11.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:10 smithi173 ceph-mon[57540]: from='client.44653 -' entity='client.iscsi.foo.smithi092.pqnhsu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-21T02:00:11.802 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-06-21T02:00:12.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:11 smithi092 ceph-mon[64276]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:12.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:11 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T02:00:12.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:11 smithi092 ceph-mon[66756]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:12.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:11 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T02:00:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:11 smithi173 ceph-mon[57540]: from='client.44770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:12.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:11 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-21T02:00:13.076 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:12 smithi092 ceph-mon[64276]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:13.076 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:12 smithi092 ceph-mon[66756]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:13.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:12 smithi173 ceph-mon[57540]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-21T02:00:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T02:00:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T02:00:14.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:13 smithi092 ceph-mon[64276]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T02:00:14.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T02:00:14.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T02:00:14.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:13 smithi092 ceph-mon[66756]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T02:00:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-21T02:00:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-21T02:00:14.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:13 smithi173 ceph-mon[57540]: from='mgr.34107 172.21.15.92:0/2606195878' entity='mgr.y' 2024-06-21T02:00:14.392 INFO:teuthology.orchestra.run.smithi092.stdout: "16.2.0", 2024-06-21T02:00:14.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:14 smithi092 ceph-mon[64276]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:14.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:14 smithi092 ceph-mon[64276]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 977 B/s rd, 0 op/s 2024-06-21T02:00:14.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:14 smithi092 ceph-mon[64276]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:14.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:14 smithi092 ceph-mon[64276]: from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:14.925 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:14 smithi092 ceph-mon[66756]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:14.925 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:14 smithi092 ceph-mon[66756]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 977 B/s rd, 0 op/s 2024-06-21T02:00:14.925 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:14 smithi092 ceph-mon[66756]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:14.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:14 smithi092 ceph-mon[66756]: from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:14.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-06-21T02:00:15.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:14 smithi173 ceph-mon[57540]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:15.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:14 smithi173 ceph-mon[57540]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 977 B/s rd, 0 op/s 2024-06-21T02:00:15.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:14 smithi173 ceph-mon[57540]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:15.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:14 smithi173 ceph-mon[57540]: from='client.44776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:17.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:16 smithi092 ceph-mon[64276]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:17.101 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:16 smithi092 ceph-mon[66756]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:17.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:16 smithi173 ceph-mon[57540]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 294 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-21T02:00:17.523 INFO:teuthology.orchestra.run.smithi092.stdout: "v16.2.2", 2024-06-21T02:00:17.523 INFO:teuthology.orchestra.run.smithi092.stdout: "v16.2.2-20210505", 2024-06-21T02:00:18.088 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:17 smithi092 ceph-mon[64276]: from='client.34857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:18.088 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:17 smithi092 ceph-mon[66756]: from='client.34857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:18.113 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-21T02:00:18.122 INFO:tasks.cephadm:Teardown begin 2024-06-21T02:00:18.122 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-21T02:00:18.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-21T02:00:18.179 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:17 smithi173 ceph-mon[57540]: from='client.34857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-21T02:00:18.186 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-21T02:00:18.186 DEBUG:teuthology.orchestra.run.smithi092:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-21T02:00:18.207 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-21T02:00:18.246 INFO:tasks.cephadm:Stopping all daemons... 2024-06-21T02:00:18.246 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-06-21T02:00:18.246 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.a 2024-06-21T02:00:18.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:18 smithi092 systemd[1]: Stopping Ceph mon.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:00:18.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:18 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a[64272]: 2024-06-21T02:00:18.570+0000 7ffa73bc3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:00:18.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:18 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a[64272]: 2024-06-21T02:00:18.570+0000 7ffa73bc3640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-06-21T02:00:18.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:18 smithi092 podman[90471]: 2024-06-21 02:00:18.693642967 +0000 UTC m=+0.249416345 container died 347e7d52844ca86ab53047fe174c28d99da1cfa0d72b21cfbd244b8734bca3b0 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0) 2024-06-21T02:00:18.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:18 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:18] ENGINE Bus STOPPING 2024-06-21T02:00:18.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:18 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-21T02:00:18.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:18 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:18] ENGINE Bus STOPPED 2024-06-21T02:00:18.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:18 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:18] ENGINE Bus STARTING 2024-06-21T02:00:19.250 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:18 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:18] ENGINE Serving on http://:::9283 2024-06-21T02:00:19.250 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:18 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:18] ENGINE Bus STARTED 2024-06-21T02:00:19.250 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:19 smithi092 podman[90471]: 2024-06-21 02:00:19.104337383 +0000 UTC m=+0.660110761 container cleanup 347e7d52844ca86ab53047fe174c28d99da1cfa0d72b21cfbd244b8734bca3b0 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240618, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2) 2024-06-21T02:00:19.250 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:19 smithi092 bash[90471]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a 2024-06-21T02:00:19.495 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.a.service' 2024-06-21T02:00:19.517 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: ::ffff:172.21.15.173 - - [21/Jun/2024:02:00:19] "GET /metrics HTTP/1.1" 200 38759 "" "Prometheus/2.33.4" 2024-06-21T02:00:19.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:19 smithi092 podman[90485]: 2024-06-21 02:00:19.355871147 +0000 UTC m=+0.661521474 container remove 347e7d52844ca86ab53047fe174c28d99da1cfa0d72b21cfbd244b8734bca3b0 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-06-21T02:00:19.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:19 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-21T02:00:19.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:19 smithi092 systemd[1]: Stopped Ceph mon.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T02:00:19.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Jun 21 02:00:19 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.a.service: Consumed 7.974s CPU time. 2024-06-21T02:00:19.546 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:00:19.547 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-06-21T02:00:19.547 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-06-21T02:00:19.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.c 2024-06-21T02:00:19.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:19 smithi092 systemd[1]: Stopping Ceph mon.c for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:00:20.264 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:19] ENGINE Bus STOPPING 2024-06-21T02:00:20.265 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c[66752]: 2024-06-21T02:00:19.900+0000 7fd6ddb02640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:00:20.265 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:19 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c[66752]: 2024-06-21T02:00:19.900+0000 7fd6ddb02640 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-06-21T02:00:20.265 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:20 smithi092 podman[90604]: 2024-06-21 02:00:20.002826653 +0000 UTC m=+0.226850154 container died 6549ab7d4df8ea76b1ae50cfe548d29200ca0a7299992d52aafc729a0fd5222d (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-21T02:00:20.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:20 smithi092 podman[90604]: 2024-06-21 02:00:20.367459197 +0000 UTC m=+0.591482699 container cleanup 6549ab7d4df8ea76b1ae50cfe548d29200ca0a7299992d52aafc729a0fd5222d (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T02:00:20.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Jun 21 02:00:20 smithi092 bash[90604]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-c 2024-06-21T02:00:20.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-21T02:00:20.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:20] ENGINE Bus STOPPED 2024-06-21T02:00:20.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:20] ENGINE Bus STARTING 2024-06-21T02:00:20.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:20] ENGINE Serving on http://:::9283 2024-06-21T02:00:20.602 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:20 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: [21/Jun/2024:02:00:20] ENGINE Bus STARTED 2024-06-21T02:00:20.845 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.c.service' 2024-06-21T02:00:20.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:00:20.879 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-06-21T02:00:20.880 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-06-21T02:00:20.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.b 2024-06-21T02:00:21.203 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:20 smithi173 systemd[1]: Stopping Ceph mon.b for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:00:21.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b[57536]: 2024-06-21T02:00:21.205+0000 7f308d3c6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:00:21.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b[57536]: 2024-06-21T02:00:21.205+0000 7f308d3c6640 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-06-21T02:00:21.478 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:21 smithi173 podman[75924]: 2024-06-21 02:00:21.441282787 +0000 UTC m=+0.356673831 container died e6b535ec7a5546c1152bd41779ca2ac70e3b843a76e95b570d26167fedb362ca (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240618) 2024-06-21T02:00:21.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:21 smithi173 podman[75924]: 2024-06-21 02:00:21.872883547 +0000 UTC m=+0.788274638 container cleanup e6b535ec7a5546c1152bd41779ca2ac70e3b843a76e95b570d26167fedb362ca (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T02:00:21.978 INFO:journalctl@ceph.mon.b.smithi173.stdout:Jun 21 02:00:21 smithi173 bash[75924]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mon-b 2024-06-21T02:00:22.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mon.b.service' 2024-06-21T02:00:22.340 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:00:22.340 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-06-21T02:00:22.340 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-06-21T02:00:22.340 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.y 2024-06-21T02:00:22.635 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:22 smithi092 systemd[1]: Stopping Ceph mgr.y for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:00:22.978 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:22 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T02:00:22.637+0000 7f99c27be640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:00:22.978 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:22 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y[59980]: 2024-06-21T02:00:22.637+0000 7f99c27be640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-06-21T02:00:22.978 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:22 smithi092 podman[90735]: 2024-06-21 02:00:22.670234033 +0000 UTC m=+0.143588645 container died fd1d3ac2265b5f453f4a961778918562145111f4be36955f249ff5ccbcfa4fe4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T02:00:23.320 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:23 smithi092 podman[90735]: 2024-06-21 02:00:23.055837203 +0000 UTC m=+0.529191815 container cleanup fd1d3ac2265b5f453f4a961778918562145111f4be36955f249ff5ccbcfa4fe4 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, ceph=True, GIT_BRANCH=HEAD) 2024-06-21T02:00:23.320 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 21 02:00:23 smithi092 bash[90735]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-y 2024-06-21T02:00:23.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.y.service' 2024-06-21T02:00:23.491 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:00:23.491 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-06-21T02:00:23.492 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-06-21T02:00:23.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.x 2024-06-21T02:00:23.838 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 02:00:23 smithi173 systemd[1]: Stopping Ceph mgr.x for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:00:24.091 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 02:00:23 smithi173 podman[76044]: 2024-06-21 02:00:23.842422043 +0000 UTC m=+0.150515532 container died 6e42ae7481a01d19551742a8b82fbd5a97ef32646983bd6036cfb2343d6cddff (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T02:00:24.454 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 02:00:24 smithi173 podman[76044]: 2024-06-21 02:00:24.194725051 +0000 UTC m=+0.502818539 container cleanup 6e42ae7481a01d19551742a8b82fbd5a97ef32646983bd6036cfb2343d6cddff (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-21T02:00:24.455 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Jun 21 02:00:24 smithi173 bash[76044]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-mgr-x 2024-06-21T02:00:24.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@mgr.x.service' 2024-06-21T02:00:24.674 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:00:24.675 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-06-21T02:00:24.675 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-21T02:00:24.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.0 2024-06-21T02:00:24.973 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:24 smithi092 systemd[1]: Stopping Ceph osd.0 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:00:25.351 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[70094]: 2024-06-21T02:00:24.975+0000 7f6a9b77a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:00:25.351 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[70094]: 2024-06-21T02:00:24.975+0000 7f6a9b77a640 -1 osd.0 156 *** Got signal Terminated *** 2024-06-21T02:00:25.351 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0[70094]: 2024-06-21T02:00:24.975+0000 7f6a9b77a640 -1 osd.0 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T02:00:30.650 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:30 smithi092 podman[90855]: 2024-06-21 02:00:30.366358586 +0000 UTC m=+5.492044748 container died 7c252b0f2dfd0299e96f8c1c98c45fbc7beec3091b1c6462878b47b9b447a809 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T02:00:30.990 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:30 smithi092 podman[90855]: 2024-06-21 02:00:30.742794917 +0000 UTC m=+5.868481077 container cleanup 7c252b0f2dfd0299e96f8c1c98c45fbc7beec3091b1c6462878b47b9b447a809 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-21T02:00:30.990 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:30 smithi092 bash[90855]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0 2024-06-21T02:00:31.310 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:30 smithi092 podman[90870]: 2024-06-21 02:00:30.994112088 +0000 UTC m=+0.620209112 container remove 7c252b0f2dfd0299e96f8c1c98c45fbc7beec3091b1c6462878b47b9b447a809 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0) 2024-06-21T02:00:31.577 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:31 smithi092 podman[90961]: 2024-06-21 02:00:31.21662127 +0000 UTC m=+0.016876811 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T02:00:31.577 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:31 smithi092 podman[90961]: 2024-06-21 02:00:31.320026145 +0000 UTC m=+0.120281681 container create 7588ffdaa8efba821e69a49dd9bd07bf7f69e4f10962902a25af12c6a0886af2 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-21T02:00:31.577 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:31 smithi092 podman[90961]: 2024-06-21 02:00:31.522409836 +0000 UTC m=+0.322665378 container init 7588ffdaa8efba821e69a49dd9bd07bf7f69e4f10962902a25af12c6a0886af2 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-06-21T02:00:31.577 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:31 smithi092 podman[90961]: 2024-06-21 02:00:31.525767333 +0000 UTC m=+0.326022865 container start 7588ffdaa8efba821e69a49dd9bd07bf7f69e4f10962902a25af12c6a0886af2 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240618, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T02:00:31.577 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:31 smithi092 podman[90961]: 2024-06-21 02:00:31.580795278 +0000 UTC m=+0.381050822 container attach 7588ffdaa8efba821e69a49dd9bd07bf7f69e4f10962902a25af12c6a0886af2 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.vendor=CentOS) 2024-06-21T02:00:31.851 INFO:journalctl@ceph.osd.0.smithi092.stdout:Jun 21 02:00:31 smithi092 podman[90961]: 2024-06-21 02:00:31.78131515 +0000 UTC m=+0.581570687 container died 7588ffdaa8efba821e69a49dd9bd07bf7f69e4f10962902a25af12c6a0886af2 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, ceph=True) 2024-06-21T02:00:32.346 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.0.service' 2024-06-21T02:00:32.391 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:00:32.392 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-21T02:00:32.392 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-21T02:00:32.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.1 2024-06-21T02:00:32.585 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:32 smithi092 systemd[1]: Stopping Ceph osd.1 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:00:32.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:32 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[72549]: 2024-06-21T02:00:32.745+0000 7fbd6c6d6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:00:32.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:32 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[72549]: 2024-06-21T02:00:32.745+0000 7fbd6c6d6640 -1 osd.1 156 *** Got signal Terminated *** 2024-06-21T02:00:32.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:32 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1[72549]: 2024-06-21T02:00:32.745+0000 7fbd6c6d6640 -1 osd.1 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T02:00:34.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:00:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:00:34.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-06-21T02:00:34.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:00:34 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:00:34.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-06-21T02:00:38.559 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:38 smithi092 podman[91064]: 2024-06-21 02:00:38.293248908 +0000 UTC m=+5.662467927 container died 9ad8cda613905109e0b5f37026dc3892f886148891938cb5f51ed41dfb55b855 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux ) 2024-06-21T02:00:38.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:38 smithi092 podman[91064]: 2024-06-21 02:00:38.670261977 +0000 UTC m=+6.039480996 container cleanup 9ad8cda613905109e0b5f37026dc3892f886148891938cb5f51ed41dfb55b855 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-06-21T02:00:38.851 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:38 smithi092 bash[91064]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1 2024-06-21T02:00:39.297 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:38 smithi092 podman[91092]: 2024-06-21 02:00:38.963524343 +0000 UTC m=+0.663354590 container remove 9ad8cda613905109e0b5f37026dc3892f886148891938cb5f51ed41dfb55b855 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-06-21T02:00:39.601 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:39 smithi092 podman[91182]: 2024-06-21 02:00:39.297900226 +0000 UTC m=+0.112174688 container create 23cfbcf9c632ec00b25e625684508bb4d844c72d77027bf208cc3923e8f38614 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0) 2024-06-21T02:00:39.601 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:39 smithi092 podman[91182]: 2024-06-21 02:00:39.202873725 +0000 UTC m=+0.017148191 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T02:00:39.601 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:39 smithi092 podman[91182]: 2024-06-21 02:00:39.516980575 +0000 UTC m=+0.331255041 container init 23cfbcf9c632ec00b25e625684508bb4d844c72d77027bf208cc3923e8f38614 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-deactivate, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-06-21T02:00:39.601 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:39 smithi092 podman[91182]: 2024-06-21 02:00:39.520530272 +0000 UTC m=+0.334804726 container start 23cfbcf9c632ec00b25e625684508bb4d844c72d77027bf208cc3923e8f38614 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-deactivate, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=squid-release-c65be81, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-06-21T02:00:40.058 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:39 smithi092 podman[91182]: 2024-06-21 02:00:39.608752746 +0000 UTC m=+0.423027200 container attach 23cfbcf9c632ec00b25e625684508bb4d844c72d77027bf208cc3923e8f38614 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-deactivate, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-21T02:00:40.058 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 21 02:00:39 smithi092 podman[91182]: 2024-06-21 02:00:39.776388443 +0000 UTC m=+0.590662902 container died 23cfbcf9c632ec00b25e625684508bb4d844c72d77027bf208cc3923e8f38614 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81) 2024-06-21T02:00:40.350 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.1.service' 2024-06-21T02:00:40.393 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:00:40.393 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-21T02:00:40.394 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-21T02:00:40.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.2 2024-06-21T02:00:40.601 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:40 smithi092 systemd[1]: Stopping Ceph osd.2 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:00:41.100 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[74968]: 2024-06-21T02:00:40.723+0000 7f6c97806640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:00:41.101 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[74968]: 2024-06-21T02:00:40.723+0000 7f6c97806640 -1 osd.2 156 *** Got signal Terminated *** 2024-06-21T02:00:41.101 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:40 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2[74968]: 2024-06-21T02:00:40.723+0000 7f6c97806640 -1 osd.2 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T02:00:44.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:00:44 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=error ts=2024-06-21T02:00:44.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-06-21T02:00:44.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:00:44 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:00:44.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-06-21T02:00:44.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:00:44 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:00:44.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-06-21T02:00:46.351 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:46 smithi092 podman[91285]: 2024-06-21 02:00:46.088341388 +0000 UTC m=+5.466355780 container died 0fe024e7dad9d0e0f5e1e50b66d27e0d673aa64b150ddccc695cea4c780b9662 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-21T02:00:46.762 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:46 smithi092 podman[91285]: 2024-06-21 02:00:46.464779253 +0000 UTC m=+5.842793646 container cleanup 0fe024e7dad9d0e0f5e1e50b66d27e0d673aa64b150ddccc695cea4c780b9662 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-21T02:00:46.762 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:46 smithi092 bash[91285]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2 2024-06-21T02:00:47.082 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:46 smithi092 podman[91301]: 2024-06-21 02:00:46.766303446 +0000 UTC m=+0.677141537 container remove 0fe024e7dad9d0e0f5e1e50b66d27e0d673aa64b150ddccc695cea4c780b9662 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T02:00:47.341 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:47 smithi092 podman[91392]: 2024-06-21 02:00:46.989227446 +0000 UTC m=+0.017557372 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T02:00:47.341 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:47 smithi092 podman[91392]: 2024-06-21 02:00:47.142342258 +0000 UTC m=+0.170672181 container create 8941deee8ce2ca03408f6e2e2513120aff65f460a8168cea5eebb86b6b5ccc25 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=squid-release-c65be81, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-21T02:00:47.341 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:47 smithi092 podman[91392]: 2024-06-21 02:00:47.344849476 +0000 UTC m=+0.373179413 container init 8941deee8ce2ca03408f6e2e2513120aff65f460a8168cea5eebb86b6b5ccc25 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-06-21T02:00:47.478 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:00:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:00:46.986Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.92:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.92:8765: connect: connection refused" 2024-06-21T02:00:47.479 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:00:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:00:46.986Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.92:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.92:8765: connect: connection refused" 2024-06-21T02:00:47.479 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:00:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:00:46.986Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.92:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.92:8765: connect: connection refused" 2024-06-21T02:00:47.479 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:00:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:00:46.986Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.92:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.92:8765: connect: connection refused" 2024-06-21T02:00:47.479 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:00:46 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:00:46.987Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.92:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.92:8765: connect: connection refused" 2024-06-21T02:00:47.601 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:47 smithi092 podman[91392]: 2024-06-21 02:00:47.348401665 +0000 UTC m=+0.376731596 container start 8941deee8ce2ca03408f6e2e2513120aff65f460a8168cea5eebb86b6b5ccc25 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-release-c65be81, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-06-21T02:00:47.601 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:47 smithi092 podman[91392]: 2024-06-21 02:00:47.461338042 +0000 UTC m=+0.489667966 container attach 8941deee8ce2ca03408f6e2e2513120aff65f460a8168cea5eebb86b6b5ccc25 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-06-21T02:00:47.860 INFO:journalctl@ceph.osd.2.smithi092.stdout:Jun 21 02:00:47 smithi092 podman[91392]: 2024-06-21 02:00:47.609017996 +0000 UTC m=+0.637347927 container died 8941deee8ce2ca03408f6e2e2513120aff65f460a8168cea5eebb86b6b5ccc25 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-2-deactivate, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-06-21T02:00:48.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.2.service' 2024-06-21T02:00:48.190 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:00:48.191 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-21T02:00:48.191 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-21T02:00:48.191 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.3 2024-06-21T02:00:48.531 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:48 smithi092 systemd[1]: Stopping Ceph osd.3 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:00:48.851 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:48 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[77410]: 2024-06-21T02:00:48.534+0000 7fb954650640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:00:48.851 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:48 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[77410]: 2024-06-21T02:00:48.534+0000 7fb954650640 -1 osd.3 156 *** Got signal Terminated *** 2024-06-21T02:00:48.851 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:48 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[77410]: 2024-06-21T02:00:48.534+0000 7fb954650640 -1 osd.3 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T02:00:51.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:00:51 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:00:51.096+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:00:51.850 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:51 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[77410]: 2024-06-21T02:00:51.393+0000 7fb94cc5e640 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:28.159044+0000 front 2024-06-21T02:00:28.159016+0000 (oldest deadline 2024-06-21T02:00:50.458933+0000) 2024-06-21T02:00:51.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:00:51 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:00:51.654+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:00:52.317 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:00:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:00:52.048+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:00:52.639 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:00:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:00:52.320+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:00:52.850 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:52 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[77410]: 2024-06-21T02:00:52.395+0000 7fb94cc5e640 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:28.159044+0000 front 2024-06-21T02:00:28.159016+0000 (oldest deadline 2024-06-21T02:00:50.458933+0000) 2024-06-21T02:00:52.981 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:52.683+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:27.458766+0000 front 2024-06-21T02:00:27.458821+0000 (oldest deadline 2024-06-21T02:00:52.158577+0000) 2024-06-21T02:00:52.981 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:00:52 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:00:52.643+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:00:53.366 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:00:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:00:53.030+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:00:53.600 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:53 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3[77410]: 2024-06-21T02:00:53.352+0000 7fb94cc5e640 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:28.159044+0000 front 2024-06-21T02:00:28.159016+0000 (oldest deadline 2024-06-21T02:00:50.458933+0000) 2024-06-21T02:00:53.685 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:00:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:00:53.370+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:00:53.978 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:53.693+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:27.458766+0000 front 2024-06-21T02:00:27.458821+0000 (oldest deadline 2024-06-21T02:00:52.158577+0000) 2024-06-21T02:00:53.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:00:53 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:00:53.689+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:00:54.265 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:53 smithi092 podman[91496]: 2024-06-21 02:00:53.993590321 +0000 UTC m=+5.568335810 container died f0817e4dfec7771d4d21b5a9347fc8005b10e886368dbf0e812685e4ec7a418f (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-06-21T02:00:54.265 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:00:54 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=error ts=2024-06-21T02:00:54.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-06-21T02:00:54.265 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:00:54 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:00:54.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-06-21T02:00:54.266 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:00:54 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:00:54.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-06-21T02:00:54.334 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:00:54 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:00:54.052+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:00:54.601 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:54 smithi092 podman[91496]: 2024-06-21 02:00:54.37593663 +0000 UTC m=+5.950682118 container cleanup f0817e4dfec7771d4d21b5a9347fc8005b10e886368dbf0e812685e4ec7a418f (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-06-21T02:00:54.601 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:54 smithi092 bash[91496]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3 2024-06-21T02:00:54.618 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:00:54 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:00:54.338+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:00:54.978 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:54 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:54.648+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:27.458766+0000 front 2024-06-21T02:00:27.458821+0000 (oldest deadline 2024-06-21T02:00:52.158577+0000) 2024-06-21T02:00:54.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:00:54 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:00:54.652+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:00:54.990 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:54 smithi092 podman[91513]: 2024-06-21 02:00:54.643857615 +0000 UTC m=+0.642686776 container remove f0817e4dfec7771d4d21b5a9347fc8005b10e886368dbf0e812685e4ec7a418f (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, GIT_CLEAN=True) 2024-06-21T02:00:55.268 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:54 smithi092 podman[91602]: 2024-06-21 02:00:54.994621178 +0000 UTC m=+0.113928657 container create ad8ac399918e9b02fbf3b0d557dab56d453e5a1e09b8c85e9144ed1e478dddfd (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-06-21T02:00:55.268 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:54 smithi092 podman[91602]: 2024-06-21 02:00:54.897783517 +0000 UTC m=+0.017090949 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T02:00:55.268 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:55 smithi092 podman[91602]: 2024-06-21 02:00:55.197264246 +0000 UTC m=+0.316571675 container init ad8ac399918e9b02fbf3b0d557dab56d453e5a1e09b8c85e9144ed1e478dddfd (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-deactivate, RELEASE=squid-release-c65be81, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T02:00:55.268 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:55 smithi092 podman[91602]: 2024-06-21 02:00:55.20067578 +0000 UTC m=+0.319983197 container start ad8ac399918e9b02fbf3b0d557dab56d453e5a1e09b8c85e9144ed1e478dddfd (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81) 2024-06-21T02:00:55.268 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:55 smithi092 podman[91602]: 2024-06-21 02:00:55.272158655 +0000 UTC m=+0.391466074 container attach ad8ac399918e9b02fbf3b0d557dab56d453e5a1e09b8c85e9144ed1e478dddfd (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-deactivate, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T02:00:55.362 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:00:55 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:00:55.073+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:00:55.601 INFO:journalctl@ceph.osd.3.smithi092.stdout:Jun 21 02:00:55 smithi092 podman[91602]: 2024-06-21 02:00:55.453032542 +0000 UTC m=+0.572339961 container died ad8ac399918e9b02fbf3b0d557dab56d453e5a1e09b8c85e9144ed1e478dddfd (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T02:00:55.633 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:00:55 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:00:55.365+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:00:55.953 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.3.service' 2024-06-21T02:00:55.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:00:55 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:00:55.636+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:00:55.978 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:55 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:55.696+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:27.458766+0000 front 2024-06-21T02:00:27.458821+0000 (oldest deadline 2024-06-21T02:00:52.158577+0000) 2024-06-21T02:00:55.991 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:00:55.991 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-21T02:00:55.991 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-21T02:00:55.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.4 2024-06-21T02:00:56.284 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:56 smithi173 systemd[1]: Stopping Ceph osd.4 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:00:56.285 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:00:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:00:56.054+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:00:56.608 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:00:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:00:56.385+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:00:56.609 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:56.288+0000 7f051a923640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:00:56.609 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:56.288+0000 7f051a923640 -1 osd.4 156 *** Got signal Terminated *** 2024-06-21T02:00:56.609 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:56.288+0000 7f051a923640 -1 osd.4 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T02:00:56.978 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:56.647+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:27.458766+0000 front 2024-06-21T02:00:27.458821+0000 (oldest deadline 2024-06-21T02:00:52.158577+0000) 2024-06-21T02:00:56.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:00:56 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:00:56.612+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:00:57.426 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:00:57 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:00:57.096+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:00:57.687 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:00:57 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:00:57.429+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:00:57.687 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:00:57 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:00:57.594+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:00:57.978 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:57 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:57.691+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:27.458766+0000 front 2024-06-21T02:00:27.458821+0000 (oldest deadline 2024-06-21T02:00:52.158577+0000) 2024-06-21T02:00:58.470 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:00:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:00:58.077+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:00:58.728 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:58.667+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:27.458766+0000 front 2024-06-21T02:00:27.458821+0000 (oldest deadline 2024-06-21T02:00:52.158577+0000) 2024-06-21T02:00:58.728 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:58.667+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:32.158923+0000 front 2024-06-21T02:00:32.159014+0000 (oldest deadline 2024-06-21T02:00:58.559444+0000) 2024-06-21T02:00:58.728 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:00:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:00:58.581+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:00:58.728 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:00:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:00:58.581+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:00:58.729 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:00:58 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:00:58.473+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:00:59.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:00:59 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:00:59.117+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:00:59.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:00:59 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:00:59.483+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:00:59.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:00:59 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:00:59.483+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:00:59.978 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:59 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:59.627+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:27.458766+0000 front 2024-06-21T02:00:27.458821+0000 (oldest deadline 2024-06-21T02:00:52.158577+0000) 2024-06-21T02:00:59.979 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:00:59 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:00:59.627+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:32.158923+0000 front 2024-06-21T02:00:32.159014+0000 (oldest deadline 2024-06-21T02:00:58.559444+0000) 2024-06-21T02:00:59.979 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:00:59 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:00:59.576+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:00:59.979 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:00:59 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:00:59.576+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:00.443 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:00 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:00.157+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:01:00.728 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:00 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:01:00.605+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:27.458766+0000 front 2024-06-21T02:00:27.458821+0000 (oldest deadline 2024-06-21T02:00:52.158577+0000) 2024-06-21T02:01:00.728 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:00 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4[60759]: 2024-06-21T02:01:00.605+0000 7f0516f39640 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:32.158923+0000 front 2024-06-21T02:00:32.159014+0000 (oldest deadline 2024-06-21T02:00:58.559444+0000) 2024-06-21T02:01:00.728 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:00 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:00.626+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:00.728 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:00 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:00.626+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:00.729 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:00 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:00.447+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:00.729 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:00 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:00.447+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:01.448 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:01 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:01.183+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:01:01.449 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:01 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:01.183+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:36.781818+0000 front 2024-06-21T02:00:36.781845+0000 (oldest deadline 2024-06-21T02:01:00.881796+0000) 2024-06-21T02:01:01.728 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:01 smithi173 podman[76192]: 2024-06-21 02:01:01.626115226 +0000 UTC m=+5.440948007 container died d42a6ffd2e665eea2feee144895412d4a82a3f96e77d4b09cbb1d21366f1eedc (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-06-21T02:01:01.728 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:01 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:01.669+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:01.729 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:01 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:01.669+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:01.729 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:01 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:01.450+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:01.729 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:01 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:01.450+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:02.228 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:02 smithi173 podman[76192]: 2024-06-21 02:01:02.080519845 +0000 UTC m=+5.895352660 container cleanup d42a6ffd2e665eea2feee144895412d4a82a3f96e77d4b09cbb1d21366f1eedc (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4, org.label-schema.build-date=20240618, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-21T02:01:02.228 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:02 smithi173 bash[76192]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4 2024-06-21T02:01:02.228 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:02.169+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:01:02.228 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:02.169+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:36.781818+0000 front 2024-06-21T02:00:36.781845+0000 (oldest deadline 2024-06-21T02:01:00.881796+0000) 2024-06-21T02:01:02.628 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:02 smithi173 podman[76223]: 2024-06-21 02:01:02.357179812 +0000 UTC m=+0.727213073 container remove d42a6ffd2e665eea2feee144895412d4a82a3f96e77d4b09cbb1d21366f1eedc (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-06-21T02:01:02.629 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:02.500+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:02.629 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:02.500+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:02.948 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:02 smithi173 podman[76311]: 2024-06-21 02:01:02.586536872 +0000 UTC m=+0.017586729 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T02:01:02.948 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:02 smithi173 podman[76311]: 2024-06-21 02:01:02.708112594 +0000 UTC m=+0.139162451 container create bad1fe1a7fec9c9192aeb947a809cfee078d89f8a843d564b3fc1f01d953374e (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-21T02:01:02.948 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:02.631+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:02.948 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:02 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:02.631+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:03.204 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:02 smithi173 podman[76311]: 2024-06-21 02:01:02.95207867 +0000 UTC m=+0.383128527 container init bad1fe1a7fec9c9192aeb947a809cfee078d89f8a843d564b3fc1f01d953374e (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-21T02:01:03.204 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:02 smithi173 podman[76311]: 2024-06-21 02:01:02.95552345 +0000 UTC m=+0.386573318 container start bad1fe1a7fec9c9192aeb947a809cfee078d89f8a843d564b3fc1f01d953374e (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618) 2024-06-21T02:01:03.204 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:03 smithi173 podman[76311]: 2024-06-21 02:01:03.018781483 +0000 UTC m=+0.449831354 container attach bad1fe1a7fec9c9192aeb947a809cfee078d89f8a843d564b3fc1f01d953374e (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81) 2024-06-21T02:01:03.204 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:03.122+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:01:03.204 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:03.122+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:36.781818+0000 front 2024-06-21T02:00:36.781845+0000 (oldest deadline 2024-06-21T02:01:00.881796+0000) 2024-06-21T02:01:03.478 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 21 02:01:03 smithi173 podman[76311]: 2024-06-21 02:01:03.207893557 +0000 UTC m=+0.638943417 container died bad1fe1a7fec9c9192aeb947a809cfee078d89f8a843d564b3fc1f01d953374e (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-06-21T02:01:03.802 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:03.607+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:03.802 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:03.607+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:03.803 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:03.541+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:03.803 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:03 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:03.541+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:03.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.4.service' 2024-06-21T02:01:03.865 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:01:03.865 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-21T02:01:03.866 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-21T02:01:03.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.5 2024-06-21T02:01:04.126 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:03 smithi173 systemd[1]: Stopping Ceph osd.5 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:01:04.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:04 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:04.128+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:01:04.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:04 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:04.128+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:36.781818+0000 front 2024-06-21T02:00:36.781845+0000 (oldest deadline 2024-06-21T02:01:00.881796+0000) 2024-06-21T02:01:04.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:04 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:04.218+0000 7fc96b3d0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:01:04.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:04 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:04.218+0000 7fc96b3d0640 -1 osd.5 156 *** Got signal Terminated *** 2024-06-21T02:01:04.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:04 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:04.218+0000 7fc96b3d0640 -1 osd.5 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T02:01:04.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:04 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=error ts=2024-06-21T02:01:04.242Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-06-21T02:01:04.602 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:04 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:01:04.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-06-21T02:01:04.602 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:04 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:01:04.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-06-21T02:01:04.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:04 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:04.645+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:04.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:04 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:04.645+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:04.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:04 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:04.553+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:04.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:04 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:04.553+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:05.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:05.094+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:01:05.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:05.094+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:36.781818+0000 front 2024-06-21T02:00:36.781845+0000 (oldest deadline 2024-06-21T02:01:00.881796+0000) 2024-06-21T02:01:05.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:05.593+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:05.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:05.593+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:05.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:05.628+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:05.979 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:05 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:05.628+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:06.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:06.116+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:01:06.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:06.116+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:36.781818+0000 front 2024-06-21T02:00:36.781845+0000 (oldest deadline 2024-06-21T02:01:00.881796+0000) 2024-06-21T02:01:06.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:06.557+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:06.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:06.557+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:06.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:06.679+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:06.979 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:06.679+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:06.979 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:06 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:06.679+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:45.575088+0000 front 2024-06-21T02:00:45.575028+0000 (oldest deadline 2024-06-21T02:01:06.674896+0000) 2024-06-21T02:01:07.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:07.127+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:01:07.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:07.127+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:36.781818+0000 front 2024-06-21T02:00:36.781845+0000 (oldest deadline 2024-06-21T02:01:00.881796+0000) 2024-06-21T02:01:07.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:07.561+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:07.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:07.561+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:07.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:07.699+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:07.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:07.699+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:07.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:07 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:07.699+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:45.575088+0000 front 2024-06-21T02:00:45.575028+0000 (oldest deadline 2024-06-21T02:01:06.674896+0000) 2024-06-21T02:01:08.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:08.101+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:01:08.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:08.101+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:36.781818+0000 front 2024-06-21T02:00:36.781845+0000 (oldest deadline 2024-06-21T02:01:00.881796+0000) 2024-06-21T02:01:08.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:08.101+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:43.182616+0000 front 2024-06-21T02:00:43.182724+0000 (oldest deadline 2024-06-21T02:01:07.882577+0000) 2024-06-21T02:01:08.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:08.744+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:08.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:08.744+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:08.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:08.744+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:45.575088+0000 front 2024-06-21T02:00:45.575028+0000 (oldest deadline 2024-06-21T02:01:06.674896+0000) 2024-06-21T02:01:08.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:08.557+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:08.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:08.557+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:08.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:08.557+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:09.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:09.116+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:29.180855+0000 front 2024-06-21T02:00:29.180783+0000 (oldest deadline 2024-06-21T02:00:50.880650+0000) 2024-06-21T02:01:09.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:09.116+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:36.781818+0000 front 2024-06-21T02:00:36.781845+0000 (oldest deadline 2024-06-21T02:01:00.881796+0000) 2024-06-21T02:01:09.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5[63053]: 2024-06-21T02:01:09.116+0000 7fc9639de640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:43.182616+0000 front 2024-06-21T02:00:43.182724+0000 (oldest deadline 2024-06-21T02:01:07.882577+0000) 2024-06-21T02:01:09.841 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:09 smithi173 podman[76413]: 2024-06-21 02:01:09.590920453 +0000 UTC m=+5.493580544 container died b6050a92252eb67ab6c60a0ba73c332b601201e9ffda0e788116a4c24e2dc8e7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-21T02:01:09.841 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:09.700+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:09.841 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:09.700+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:09.841 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:09.700+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:45.575088+0000 front 2024-06-21T02:00:45.575028+0000 (oldest deadline 2024-06-21T02:01:06.674896+0000) 2024-06-21T02:01:09.842 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:09.568+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:09.842 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:09.568+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:09.842 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:09 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:09.568+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:10.209 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:10 smithi173 podman[76413]: 2024-06-21 02:01:10.019766403 +0000 UTC m=+5.922426486 container cleanup b6050a92252eb67ab6c60a0ba73c332b601201e9ffda0e788116a4c24e2dc8e7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-06-21T02:01:10.209 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:10 smithi173 bash[76413]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5 2024-06-21T02:01:10.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:10 smithi173 podman[76428]: 2024-06-21 02:01:10.213604248 +0000 UTC m=+0.618184450 container remove b6050a92252eb67ab6c60a0ba73c332b601201e9ffda0e788116a4c24e2dc8e7 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5, RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240618, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-21T02:01:10.808 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:10 smithi173 podman[76519]: 2024-06-21 02:01:10.540840198 +0000 UTC m=+0.017460875 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T02:01:10.808 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:10 smithi173 podman[76519]: 2024-06-21 02:01:10.672857282 +0000 UTC m=+0.149477961 container create 56f34e1a85f214630098586aad4774b4b1268bdd74415c29a065bdb91a96d200 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-06-21T02:01:10.808 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:10.700+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:10.808 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:10.700+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:10.808 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:10.700+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:45.575088+0000 front 2024-06-21T02:00:45.575028+0000 (oldest deadline 2024-06-21T02:01:06.674896+0000) 2024-06-21T02:01:10.809 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:10.548+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:10.809 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:10.548+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:10.809 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:10 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:10.548+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:11.158 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:10 smithi173 podman[76519]: 2024-06-21 02:01:10.900667394 +0000 UTC m=+0.377288082 container init 56f34e1a85f214630098586aad4774b4b1268bdd74415c29a065bdb91a96d200 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.build-date=20240618, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-06-21T02:01:11.159 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:10 smithi173 podman[76519]: 2024-06-21 02:01:10.904561279 +0000 UTC m=+0.381181958 container start 56f34e1a85f214630098586aad4774b4b1268bdd74415c29a065bdb91a96d200 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-release-c65be81, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-06-21T02:01:11.159 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:10 smithi173 podman[76519]: 2024-06-21 02:01:10.983593872 +0000 UTC m=+0.460214563 container attach 56f34e1a85f214630098586aad4774b4b1268bdd74415c29a065bdb91a96d200 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240618, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-06-21T02:01:11.478 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 21 02:01:11 smithi173 podman[76519]: 2024-06-21 02:01:11.163045479 +0000 UTC m=+0.639666167 container died 56f34e1a85f214630098586aad4774b4b1268bdd74415c29a065bdb91a96d200 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-5-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240618, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-21T02:01:11.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.5.service' 2024-06-21T02:01:11.730 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:01:11.730 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-21T02:01:11.730 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-06-21T02:01:11.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.6 2024-06-21T02:01:11.770 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:11.504+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:11.770 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:11.504+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:11.770 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:11.504+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:11.771 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:11.716+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:11.771 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:11.716+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:11.771 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:11 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:11.716+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:45.575088+0000 front 2024-06-21T02:00:45.575028+0000 (oldest deadline 2024-06-21T02:01:06.674896+0000) 2024-06-21T02:01:12.053 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:11 smithi173 systemd[1]: Stopping Ceph osd.6 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:01:12.478 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:12.055+0000 7fb8298f7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:01:12.478 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:12.055+0000 7fb8298f7640 -1 osd.6 156 *** Got signal Terminated *** 2024-06-21T02:01:12.478 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:12.055+0000 7fb8298f7640 -1 osd.6 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T02:01:12.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:12.749+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:12.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:12.749+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:12.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:12.749+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:45.575088+0000 front 2024-06-21T02:00:45.575028+0000 (oldest deadline 2024-06-21T02:01:06.674896+0000) 2024-06-21T02:01:12.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:12.531+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:12.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:12.531+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:12.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:12 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:12.531+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:13.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:13.706+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:13.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:13.706+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:13.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:13.706+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:45.575088+0000 front 2024-06-21T02:00:45.575028+0000 (oldest deadline 2024-06-21T02:01:06.674896+0000) 2024-06-21T02:01:13.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:13.577+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:13.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:13.577+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:13.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:13 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:13.577+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:14.512 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:14 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=error ts=2024-06-21T02:01:14.243Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-06-21T02:01:14.512 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:14 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:01:14.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-06-21T02:01:14.512 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:14 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:01:14.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-06-21T02:01:14.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:14 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:14.704+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:14.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:14 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:14.704+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:14.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:14 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:14.704+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:45.575088+0000 front 2024-06-21T02:00:45.575028+0000 (oldest deadline 2024-06-21T02:01:06.674896+0000) 2024-06-21T02:01:14.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:14 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:14.576+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:14.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:14 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:14.576+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:14.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:14 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:14.576+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:15.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:15.723+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:15.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:15.723+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:15.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:15.723+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:45.575088+0000 front 2024-06-21T02:00:45.575028+0000 (oldest deadline 2024-06-21T02:01:06.674896+0000) 2024-06-21T02:01:15.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:15.723+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:50.676323+0000 front 2024-06-21T02:00:50.676285+0000 (oldest deadline 2024-06-21T02:01:14.776172+0000) 2024-06-21T02:01:15.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:15.602+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:15.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:15.602+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:15.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:15.602+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:15.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:15 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:15.602+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:53.142548+0000 front 2024-06-21T02:00:53.142483+0000 (oldest deadline 2024-06-21T02:01:15.342794+0000) 2024-06-21T02:01:16.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:16.735+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.471375+0000 front 2024-06-21T02:00:26.471412+0000 (oldest deadline 2024-06-21T02:00:51.171402+0000) 2024-06-21T02:01:16.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:16.735+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:35.673400+0000 front 2024-06-21T02:00:35.673484+0000 (oldest deadline 2024-06-21T02:00:58.573370+0000) 2024-06-21T02:01:16.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:16.735+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:45.575088+0000 front 2024-06-21T02:00:45.575028+0000 (oldest deadline 2024-06-21T02:01:06.674896+0000) 2024-06-21T02:01:16.978 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6[65342]: 2024-06-21T02:01:16.735+0000 7fb821f05640 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:50.676323+0000 front 2024-06-21T02:00:50.676285+0000 (oldest deadline 2024-06-21T02:01:14.776172+0000) 2024-06-21T02:01:16.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:16.587+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:16.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:16.588+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:16.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:16.588+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:16.979 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:16 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:16.588+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:53.142548+0000 front 2024-06-21T02:00:53.142483+0000 (oldest deadline 2024-06-21T02:01:15.342794+0000) 2024-06-21T02:01:17.728 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:17 smithi173 podman[76620]: 2024-06-21 02:01:17.453758056 +0000 UTC m=+5.500045956 container died dffac360ac20aabb4ccf2443b0f04daa47db3a0184c9855e1471b55dfb0fd69c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6, ceph=True, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-06-21T02:01:17.729 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:17 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:17.621+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:17.729 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:17 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:17.621+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:17.729 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:17 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:17.621+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:17.729 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:17 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:17.621+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:53.142548+0000 front 2024-06-21T02:00:53.142483+0000 (oldest deadline 2024-06-21T02:01:15.342794+0000) 2024-06-21T02:01:18.070 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:17 smithi173 podman[76620]: 2024-06-21 02:01:17.906685557 +0000 UTC m=+5.952973456 container cleanup dffac360ac20aabb4ccf2443b0f04daa47db3a0184c9855e1471b55dfb0fd69c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6, ceph=True, GIT_CLEAN=True, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240618, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-21T02:01:18.071 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:17 smithi173 bash[76620]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6 2024-06-21T02:01:18.478 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:18 smithi173 podman[76636]: 2024-06-21 02:01:18.074688087 +0000 UTC m=+0.613575175 container remove dffac360ac20aabb4ccf2443b0f04daa47db3a0184c9855e1471b55dfb0fd69c (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240618, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-21T02:01:18.777 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:18.609+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:18.777 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:18.609+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:18.778 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:18.609+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:18.778 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:18 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:18.609+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:53.142548+0000 front 2024-06-21T02:00:53.142483+0000 (oldest deadline 2024-06-21T02:01:15.342794+0000) 2024-06-21T02:01:18.778 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:18 smithi173 podman[76727]: 2024-06-21 02:01:18.431465247 +0000 UTC m=+0.017569904 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T02:01:18.778 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:18 smithi173 podman[76727]: 2024-06-21 02:01:18.533878001 +0000 UTC m=+0.119982648 container create f09e7b7e0ca8c0fdd138c331f22e7419fef90ad47c0a9cd7cc7c1430ea957702 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, ceph=True) 2024-06-21T02:01:18.778 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:18 smithi173 podman[76727]: 2024-06-21 02:01:18.778276571 +0000 UTC m=+0.364381223 container init f09e7b7e0ca8c0fdd138c331f22e7419fef90ad47c0a9cd7cc7c1430ea957702 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, ceph=True) 2024-06-21T02:01:18.778 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:18 smithi173 podman[76727]: 2024-06-21 02:01:18.78160967 +0000 UTC m=+0.367714318 container start f09e7b7e0ca8c0fdd138c331f22e7419fef90ad47c0a9cd7cc7c1430ea957702 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD) 2024-06-21T02:01:19.036 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:18 smithi173 podman[76727]: 2024-06-21 02:01:18.878492314 +0000 UTC m=+0.464596963 container attach f09e7b7e0ca8c0fdd138c331f22e7419fef90ad47c0a9cd7cc7c1430ea957702 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-release-c65be81, GIT_CLEAN=True, org.label-schema.build-date=20240618, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-21T02:01:19.319 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:19 smithi173 podman[76727]: 2024-06-21 02:01:19.040622631 +0000 UTC m=+0.626727282 container died f09e7b7e0ca8c0fdd138c331f22e7419fef90ad47c0a9cd7cc7c1430ea957702 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-06-21T02:01:19.570 INFO:journalctl@ceph.osd.6.smithi173.stdout:Jun 21 02:01:19 smithi173 podman[76750]: 2024-06-21 02:01:19.565261449 +0000 UTC m=+0.519035242 container remove f09e7b7e0ca8c0fdd138c331f22e7419fef90ad47c0a9cd7cc7c1430ea957702 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240618) 2024-06-21T02:01:19.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.6.service' 2024-06-21T02:01:19.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:01:19.616 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-06-21T02:01:19.616 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-06-21T02:01:19.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.7 2024-06-21T02:01:19.956 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:19 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:19.628+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:19.956 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:19 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:19.628+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:19.956 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:19 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:19.628+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:19.956 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:19 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:19.628+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:53.142548+0000 front 2024-06-21T02:00:53.142483+0000 (oldest deadline 2024-06-21T02:01:15.342794+0000) 2024-06-21T02:01:19.956 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:19 smithi173 systemd[1]: Stopping Ceph osd.7 for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:01:19.956 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:19 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:19.959+0000 7f496aeb6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-21T02:01:19.956 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:19 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:19.959+0000 7f496aeb6640 -1 osd.7 156 *** Got signal Terminated *** 2024-06-21T02:01:19.956 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:19 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:19.959+0000 7f496aeb6640 -1 osd.7 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-21T02:01:20.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:20 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:20.590+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:20.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:20 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:20.590+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:20.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:20 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:20.590+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:20.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:20 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:20.590+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:53.142548+0000 front 2024-06-21T02:00:53.142483+0000 (oldest deadline 2024-06-21T02:01:15.342794+0000) 2024-06-21T02:01:21.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:21.544+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:21.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:21.544+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:21.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:21.544+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:21.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:21 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:21.544+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:53.142548+0000 front 2024-06-21T02:00:53.142483+0000 (oldest deadline 2024-06-21T02:01:15.342794+0000) 2024-06-21T02:01:22.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:22.517+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:22.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:22.517+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:22.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:22.517+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:22.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:22.517+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:53.142548+0000 front 2024-06-21T02:00:53.142483+0000 (oldest deadline 2024-06-21T02:01:15.342794+0000) 2024-06-21T02:01:22.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:22 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:22.517+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.173:6804 osd.4 since back 2024-06-21T02:00:57.544045+0000 front 2024-06-21T02:00:57.543907+0000 (oldest deadline 2024-06-21T02:01:21.643841+0000) 2024-06-21T02:01:23.880 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:23 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:23.562+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:23.880 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:23 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:23.562+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:23.880 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:23 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:23.562+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:23.880 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:23 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:23.562+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:53.142548+0000 front 2024-06-21T02:00:53.142483+0000 (oldest deadline 2024-06-21T02:01:15.342794+0000) 2024-06-21T02:01:23.880 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:23 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:23.562+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.173:6804 osd.4 since back 2024-06-21T02:00:57.544045+0000 front 2024-06-21T02:00:57.543907+0000 (oldest deadline 2024-06-21T02:01:21.643841+0000) 2024-06-21T02:01:24.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=error ts=2024-06-21T02:01:24.244Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-06-21T02:01:24.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:01:24.245Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi092.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.92:8443: connect: connection refused" 2024-06-21T02:01:24.601 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:24 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=warn ts=2024-06-21T02:01:24.245Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-06-21T02:01:24.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:24 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:24.565+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-06-21T02:00:26.139954+0000 front 2024-06-21T02:00:26.139917+0000 (oldest deadline 2024-06-21T02:00:51.439796+0000) 2024-06-21T02:01:24.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:24 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:24.565+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-06-21T02:00:34.340718+0000 front 2024-06-21T02:00:34.340790+0000 (oldest deadline 2024-06-21T02:00:59.040631+0000) 2024-06-21T02:01:24.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:24 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:24.565+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-06-21T02:00:41.941606+0000 front 2024-06-21T02:00:41.941667+0000 (oldest deadline 2024-06-21T02:01:07.841577+0000) 2024-06-21T02:01:24.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:24 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:24.565+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-06-21T02:00:53.142548+0000 front 2024-06-21T02:00:53.142483+0000 (oldest deadline 2024-06-21T02:01:15.342794+0000) 2024-06-21T02:01:24.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:24 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7[67649]: 2024-06-21T02:01:24.565+0000 7f49634c4640 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.173:6804 osd.4 since back 2024-06-21T02:00:57.544045+0000 front 2024-06-21T02:00:57.543907+0000 (oldest deadline 2024-06-21T02:01:21.643841+0000) 2024-06-21T02:01:25.756 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:25 smithi173 podman[76830]: 2024-06-21 02:01:25.505717607 +0000 UTC m=+5.656207847 container died 3d835c4a13e683d9933f6eebc80045c8e97f91e81fe7d2f2358a54e9616ccbdd (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81, org.label-schema.schema-version=1.0) 2024-06-21T02:01:26.006 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:25 smithi173 podman[76830]: 2024-06-21 02:01:25.851181003 +0000 UTC m=+6.001671237 container cleanup 3d835c4a13e683d9933f6eebc80045c8e97f91e81fe7d2f2358a54e9616ccbdd (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=squid-release-c65be81, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240618, ceph=True) 2024-06-21T02:01:26.007 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:25 smithi173 bash[76830]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7 2024-06-21T02:01:26.412 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:26 smithi173 podman[76857]: 2024-06-21 02:01:26.010911521 +0000 UTC m=+0.503935605 container remove 3d835c4a13e683d9933f6eebc80045c8e97f91e81fe7d2f2358a54e9616ccbdd (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7, org.label-schema.build-date=20240618, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-06-21T02:01:26.685 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:26 smithi173 podman[76943]: 2024-06-21 02:01:26.319526164 +0000 UTC m=+0.017505983 image pull e9bfc9e474e5aa1c9fe2590308cf585a9ebc8ed3f9cf85835780aeab9836d1d3 quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-21T02:01:26.685 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:26 smithi173 podman[76943]: 2024-06-21 02:01:26.428362213 +0000 UTC m=+0.126342019 container create 3e42e3217ada656cdb0fa0d7780e48cfb7d9427906f7d724c1d404be88157f74 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-release-c65be81, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-06-21T02:01:26.685 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:26 smithi173 podman[76943]: 2024-06-21 02:01:26.605932911 +0000 UTC m=+0.303912717 container init 3e42e3217ada656cdb0fa0d7780e48cfb7d9427906f7d724c1d404be88157f74 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-deactivate, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240618, RELEASE=squid-release-c65be81) 2024-06-21T02:01:26.685 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:26 smithi173 podman[76943]: 2024-06-21 02:01:26.609140842 +0000 UTC m=+0.307120651 container start 3e42e3217ada656cdb0fa0d7780e48cfb7d9427906f7d724c1d404be88157f74 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-release-c65be81, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-21T02:01:26.685 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:26 smithi173 podman[76943]: 2024-06-21 02:01:26.688935046 +0000 UTC m=+0.386914861 container attach 3e42e3217ada656cdb0fa0d7780e48cfb7d9427906f7d724c1d404be88157f74 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-deactivate, RELEASE=squid-release-c65be81, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240618, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS) 2024-06-21T02:01:26.978 INFO:journalctl@ceph.osd.7.smithi173.stdout:Jun 21 02:01:26 smithi173 podman[76943]: 2024-06-21 02:01:26.871268273 +0000 UTC m=+0.569248088 container died 3e42e3217ada656cdb0fa0d7780e48cfb7d9427906f7d724c1d404be88157f74 (image=quay.ceph.io/ceph-ci/ceph:c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240618, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-release-c65be81, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-21T02:01:27.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@osd.7.service' 2024-06-21T02:01:27.546 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:01:27.547 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-06-21T02:01:27.547 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-06-21T02:01:27.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@prometheus.a 2024-06-21T02:01:27.733 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 systemd[1]: Stopping Ceph prometheus.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:01:28.001 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.737Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-21T02:01:28.001 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.737Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-06-21T02:01:28.001 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.737Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-06-21T02:01:28.001 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.737Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-06-21T02:01:28.001 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.737Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-06-21T02:01:28.001 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.737Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-06-21T02:01:28.001 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.737Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-21T02:01:28.002 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.737Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-06-21T02:01:28.002 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.737Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-06-21T02:01:28.002 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.738Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-06-21T02:01:28.002 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.738Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-06-21T02:01:28.002 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a[53597]: ts=2024-06-21T02:01:27.738Z caller=main.go:1066 level=info msg="See you next time!" 2024-06-21T02:01:28.002 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:27 smithi173 podman[77019]: 2024-06-21 02:01:27.747624124 +0000 UTC m=+0.105033927 container died 8e7ba220b26dccc5f14aa0d5d9628e72630745d2d847e428b52499c3bc4b0a0f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T02:01:28.375 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:28 smithi173 podman[77019]: 2024-06-21 02:01:28.111889689 +0000 UTC m=+0.469299491 container cleanup 8e7ba220b26dccc5f14aa0d5d9628e72630745d2d847e428b52499c3bc4b0a0f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-21T02:01:28.375 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:28 smithi173 bash[77019]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus-a 2024-06-21T02:01:28.375 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Jun 21 02:01:28 smithi173 bash[77047]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-prometheus.a" found: no such container 2024-06-21T02:01:28.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@prometheus.a.service' 2024-06-21T02:01:28.416 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-21T02:01:28.416 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-06-21T02:01:28.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-21T02:01:30.279 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 02:01:29 smithi092 systemd[1]: Stopping Ceph node-exporter.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:01:30.279 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 02:01:30 smithi092 podman[91977]: 2024-06-21 02:01:30.035130162 +0000 UTC m=+0.119564433 container died 099deaf4585360d46cc25fd5c6f76be6c453d08794386d277e08bcc479406836 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T02:01:30.529 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 02:01:30 smithi092 podman[91977]: 2024-06-21 02:01:30.373947174 +0000 UTC m=+0.458381444 container cleanup 099deaf4585360d46cc25fd5c6f76be6c453d08794386d277e08bcc479406836 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T02:01:30.529 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 02:01:30 smithi092 bash[91977]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a 2024-06-21T02:01:30.851 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:30 smithi092 systemd[1]: Stopping Ceph alertmanager.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:01:30.851 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 02:01:30 smithi092 podman[91991]: 2024-06-21 02:01:30.533566 +0000 UTC m=+0.498244591 container remove 099deaf4585360d46cc25fd5c6f76be6c453d08794386d277e08bcc479406836 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-21T02:01:30.851 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 02:01:30 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-06-21T02:01:30.851 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 02:01:30 smithi092 bash[92003]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter.a" found: no such container 2024-06-21T02:01:30.851 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 02:01:30 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-06-21T02:01:30.851 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 02:01:30 smithi092 systemd[1]: Stopped Ceph node-exporter.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T02:01:30.852 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Jun 21 02:01:30 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.a.service: Consumed 3.822s CPU time. 2024-06-21T02:01:31.183 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:30 smithi092 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a[58873]: level=info ts=2024-06-21T02:01:30.861Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-06-21T02:01:31.183 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:30 smithi092 podman[92053]: 2024-06-21 02:01:30.913247767 +0000 UTC m=+0.142750652 container died 9b53017582564c2343a251ead09c1c1aa111d04b9025ce042ef71a9c89945684 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T02:01:31.542 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:31 smithi092 podman[92053]: 2024-06-21 02:01:31.287110611 +0000 UTC m=+0.516613493 container cleanup 9b53017582564c2343a251ead09c1c1aa111d04b9025ce042ef71a9c89945684 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T02:01:31.543 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:31 smithi092 bash[92053]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a 2024-06-21T02:01:31.543 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:31 smithi092 bash[92080]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager.a" found: no such container 2024-06-21T02:01:31.829 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:31 smithi092 podman[92064]: 2024-06-21 02:01:31.546808475 +0000 UTC m=+0.625224917 container remove 9b53017582564c2343a251ead09c1c1aa111d04b9025ce042ef71a9c89945684 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-21T02:01:31.830 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:31 smithi092 podman[92064]: 2024-06-21 02:01:31.630898775 +0000 UTC m=+0.709315217 volume remove d91a3846db2d687ba1a0c2cbb1f01fdb2f0df7450cda0e1cdec23dbf7cd9897c 2024-06-21T02:01:31.830 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:31 smithi092 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-06-21T02:01:31.830 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Jun 21 02:01:31 smithi092 systemd[1]: Stopped Ceph alertmanager.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T02:02:05.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-21T02:02:07.311 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 02:02:07 smithi173 systemd[1]: Stopping Ceph node-exporter.b for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:02:07.600 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 02:02:07 smithi173 podman[77387]: 2024-06-21 02:02:07.314414965 +0000 UTC m=+0.113031791 container died e0cf70cfc7524c2f5bbd6b6faae31b8cb6cd895fc43c73053e05887369e42593 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T02:02:07.901 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 02:02:07 smithi173 podman[77387]: 2024-06-21 02:02:07.770820475 +0000 UTC m=+0.569437353 container cleanup e0cf70cfc7524c2f5bbd6b6faae31b8cb6cd895fc43c73053e05887369e42593 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T02:02:07.901 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 02:02:07 smithi173 bash[77387]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b 2024-06-21T02:02:08.221 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 02:02:08 smithi173 systemd[1]: Stopping Ceph grafana.a for 22eef780-2f6f-11ef-bc9d-c7b262605968... 2024-06-21T02:02:08.221 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 02:02:07 smithi173 podman[77399]: 2024-06-21 02:02:07.905385924 +0000 UTC m=+0.594887803 container remove e0cf70cfc7524c2f5bbd6b6faae31b8cb6cd895fc43c73053e05887369e42593 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-21T02:02:08.221 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 02:02:07 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-06-21T02:02:08.222 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 02:02:07 smithi173 bash[77416]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-node-exporter.b" found: no such container 2024-06-21T02:02:08.222 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 02:02:07 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-06-21T02:02:08.222 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 02:02:07 smithi173 systemd[1]: Stopped Ceph node-exporter.b for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T02:02:08.222 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Jun 21 02:02:07 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@node-exporter.b.service: Consumed 3.706s CPU time. 2024-06-21T02:02:08.478 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 02:02:08 smithi173 ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a[50914]: t=2024-06-21T02:02:08+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-06-21T02:02:08.478 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 02:02:08 smithi173 podman[77468]: 2024-06-21 02:02:08.235364491 +0000 UTC m=+0.105999706 container died 0003e9fdc5d3cb5a3bb9c6557e732af0a99327ac3a4d2f1a377a4ba9cc04371f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.component=ubi8-container, release=236.1648460182, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, distribution-scope=public, description=Ceph Grafana Container, architecture=x86_64) 2024-06-21T02:02:08.772 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 02:02:08 smithi173 podman[77468]: 2024-06-21 02:02:08.57455194 +0000 UTC m=+0.445187156 container cleanup 0003e9fdc5d3cb5a3bb9c6557e732af0a99327ac3a4d2f1a377a4ba9cc04371f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, release=236.1648460182, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=) 2024-06-21T02:02:08.772 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 02:02:08 smithi173 bash[77468]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a 2024-06-21T02:02:08.772 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 02:02:08 smithi173 podman[77482]: 2024-06-21 02:02:08.742609244 +0000 UTC m=+0.508524656 container remove 0003e9fdc5d3cb5a3bb9c6557e732af0a99327ac3a4d2f1a377a4ba9cc04371f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana-a, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, vendor=Red Hat, Inc., io.openshift.expose-services=, io.openshift.tags=base rhel8, distribution-scope=public, com.redhat.component=ubi8-container, description=Ceph Grafana Container, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8) 2024-06-21T02:02:08.772 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 02:02:08 smithi173 bash[77494]: Error: no container with name or ID "ceph-22eef780-2f6f-11ef-bc9d-c7b262605968-grafana.a" found: no such container 2024-06-21T02:02:09.225 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 02:02:08 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@grafana.a.service: Deactivated successfully. 2024-06-21T02:02:09.225 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 02:02:08 smithi173 systemd[1]: Stopped Ceph grafana.a for 22eef780-2f6f-11ef-bc9d-c7b262605968. 2024-06-21T02:02:09.225 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Jun 21 02:02:08 smithi173 systemd[1]: ceph-22eef780-2f6f-11ef-bc9d-c7b262605968@grafana.a.service: Consumed 4.123s CPU time. 2024-06-21T02:02:32.047 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-21T02:02:32.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-21T02:02:32.108 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-21T02:02:32.110 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi/7765946/remote/smithi092/crash 2024-06-21T02:02:32.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/crash -- . 2024-06-21T02:02:32.166 INFO:teuthology.orchestra.run.smithi092.stderr:tar: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-21T02:02:32.167 INFO:teuthology.orchestra.run.smithi092.stderr:tar: Error is not recoverable: exiting now 2024-06-21T02:02:32.168 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi/7765946/remote/smithi173/crash 2024-06-21T02:02:32.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/crash -- . 2024-06-21T02:02:32.205 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-21T02:02:32.205 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-06-21T02:02:32.206 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-21T02:02:32.206 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-06-21T02:02:32.235 INFO:teuthology.orchestra.run.smithi092.stdout:2024-06-21T01:51:42.781372+0000 mon.a (mon.0) 954 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-21T02:02:32.235 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-21T02:02:32.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[SEC\]' /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-06-21T02:02:32.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]' /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-06-21T02:02:32.373 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[WRN\]' /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-06-21T02:02:32.441 INFO:teuthology.orchestra.run.smithi092.stdout:2024-06-21T01:51:42.781372+0000 mon.a (mon.0) 954 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-21T02:02:32.442 INFO:tasks.cephadm:Compressing logs... 2024-06-21T02:02:32.442 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-21T02:02:32.484 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-21T02:02:32.506 INFO:teuthology.orchestra.run.smithi092.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-21T02:02:32.520 INFO:teuthology.orchestra.run.smithi092.stderr:: No such file or directory 2024-06-21T02:02:32.521 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.foo.smithi092.lzbkvh.log 2024-06-21T02:02:32.521 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.log 2024-06-21T02:02:32.529 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.foo.smithi092.lzbkvh.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.2.log 2024-06-21T02:02:32.530 INFO:teuthology.orchestra.run.smithi092.stderr: 76.3% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.foo.smithi092.lzbkvh.log.gz 2024-06-21T02:02:32.530 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.audit.log 2024-06-21T02:02:32.530 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.3.log 2024-06-21T02:02:32.530 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.1.log 2024-06-21T02:02:32.530 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mgr.y.log 2024-06-21T02:02:32.530 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.0.log 2024-06-21T02:02:32.530 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mgr.y.log: 92.8% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.log.gz 2024-06-21T02:02:32.530 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mon.c.log 2024-06-21T02:02:32.530 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.0.log: 94.2% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.audit.log.gz 2024-06-21T02:02:32.530 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mon.a.log 2024-06-21T02:02:32.530 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mon.c.log: 91.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-21T02:02:32.531 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.cephadm.log 2024-06-21T02:02:32.531 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/tcmu-runner.log 2024-06-21T02:02:32.532 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.cephadm.log: 90.7% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.cephadm.log.gz 2024-06-21T02:02:32.533 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.log 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.audit.log 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.7.log 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.5.log 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.4.log 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.7.log: /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.smpl.smithi173.khwpnd.log 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.foo.smithi173.hvgnvt.log 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.smpl.smithi173.khwpnd.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mgr.x.log 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr: 77.0% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.smpl.smithi173.khwpnd.log.gz 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.foo.smithi173.hvgnvt.log: 87.1% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.log.gz 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr: 76.3% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.foo.smithi173.hvgnvt.log.gz 2024-06-21T02:02:32.534 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.6.log 2024-06-21T02:02:32.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.cephadm.log 2024-06-21T02:02:32.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-volume.log 2024-06-21T02:02:32.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.cephadm.log: 90.5% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.audit.log.gz 2024-06-21T02:02:32.535 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mon.b.log 2024-06-21T02:02:32.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-volume.log: 83.7% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph.cephadm.log.gz 2024-06-21T02:02:32.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-volume.log.gz 2024-06-21T02:02:32.535 INFO:teuthology.orchestra.run.smithi173.stderr: 90.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-21T02:02:32.543 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-volume.log 2024-06-21T02:02:32.543 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/tcmu-runner.log: 84.2% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/tcmu-runner.log.gz 2024-06-21T02:02:32.556 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.smpl.smithi092.bxxcjp.log 2024-06-21T02:02:32.576 INFO:teuthology.orchestra.run.smithi173.stderr: 89.7% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mgr.x.log.gz 2024-06-21T02:02:32.580 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-volume.log: 91.0%/var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.smpl.smithi092.bxxcjp.log: -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-volume.log.gz 2024-06-21T02:02:32.582 INFO:teuthology.orchestra.run.smithi092.stderr: 78.2% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-client.rgw.smpl.smithi092.bxxcjp.log.gz 2024-06-21T02:02:32.798 INFO:teuthology.orchestra.run.smithi173.stderr: 92.4% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mon.b.log.gz 2024-06-21T02:02:32.884 INFO:teuthology.orchestra.run.smithi092.stderr: 88.7% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mgr.y.log.gz 2024-06-21T02:02:32.972 INFO:teuthology.orchestra.run.smithi092.stderr: 92.1% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mon.c.log.gz 2024-06-21T02:02:33.456 INFO:teuthology.orchestra.run.smithi092.stderr: 91.1% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-mon.a.log.gz 2024-06-21T02:02:33.536 INFO:teuthology.orchestra.run.smithi173.stderr: 93.6% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.6.log.gz 2024-06-21T02:02:33.673 INFO:teuthology.orchestra.run.smithi092.stderr: 93.5% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.2.log.gz 2024-06-21T02:02:33.847 INFO:teuthology.orchestra.run.smithi173.stderr: 94.0% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.7.log.gz 2024-06-21T02:02:33.963 INFO:teuthology.orchestra.run.smithi173.stderr: 93.8% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.5.log.gz 2024-06-21T02:02:34.152 INFO:teuthology.orchestra.run.smithi092.stderr: 93.8% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.0.log.gz 2024-06-21T02:02:34.158 INFO:teuthology.orchestra.run.smithi173.stderr: 93.9% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.4.log.gz 2024-06-21T02:02:34.160 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-06-21T02:02:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m1.661s 2024-06-21T02:02:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m5.625s 2024-06-21T02:02:34.160 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m0.241s 2024-06-21T02:02:34.279 INFO:teuthology.orchestra.run.smithi092.stderr: 93.8% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.1.log.gz 2024-06-21T02:02:34.387 INFO:teuthology.orchestra.run.smithi092.stderr: 93.8% -- replaced with /var/log/ceph/22eef780-2f6f-11ef-bc9d-c7b262605968/ceph-osd.3.log.gz 2024-06-21T02:02:34.390 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-06-21T02:02:34.390 INFO:teuthology.orchestra.run.smithi092.stderr:real 0m1.894s 2024-06-21T02:02:34.390 INFO:teuthology.orchestra.run.smithi092.stderr:user 0m7.977s 2024-06-21T02:02:34.390 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m0.269s 2024-06-21T02:02:34.390 INFO:tasks.cephadm:Archiving logs... 2024-06-21T02:02:34.390 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi/7765946/remote/smithi092/log 2024-06-21T02:02:34.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-21T02:02:35.421 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi/7765946/remote/smithi173/log 2024-06-21T02:02:35.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-21T02:02:36.231 INFO:tasks.cephadm:Removing cluster... 2024-06-21T02:02:36.231 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 --force 2024-06-21T02:02:36.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22eef780-2f6f-11ef-bc9d-c7b262605968 --force 2024-06-21T02:02:36.801 INFO:tasks.cephadm:Removing cephadm ... 2024-06-21T02:02:36.802 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-21T02:02:36.817 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-21T02:02:36.834 INFO:tasks.cephadm:Teardown complete 2024-06-21T02:02:36.834 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-21T02:02:36.843 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-21T02:02:36.845 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-21T02:02:36.861 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-21T02:02:36.873 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-06-21T02:02:36.878 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-21T02:02:36.879 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-06-21T02:02:36.879 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv01.front.sepia.ceph.com 3 6 377 10 +686us[ +691us] +/- 43ms 2024-06-21T02:02:36.879 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 70 -468us[ -463us] +/- 77ms 2024-06-21T02:02:36.879 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 7 -439us[ -439us] +/- 79ms 2024-06-21T02:02:36.879 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-21T02:02:36.889 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-06-21T02:02:36.893 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-21T02:02:36.893 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-06-21T02:02:36.893 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv01.front.sepia.ceph.com 3 6 377 6 +628us[ +633us] +/- 43ms 2024-06-21T02:02:36.893 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 -557us[ -552us] +/- 78ms 2024-06-21T02:02:36.893 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 -521us[ -517us] +/- 79ms 2024-06-21T02:02:36.893 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-21T02:02:36.894 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-21T02:02:36.905 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-21T02:02:36.906 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-21T02:02:36.950 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-21T02:02:37.027 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-21T02:02:37.100 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-21T02:02:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-21T02:02:37.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-21T02:02:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-21T02:02:37.158 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-21T02:02:37.168 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-21T02:02:37.190 INFO:teuthology.task.internal:Duration was 1641.548155 seconds 2024-06-21T02:02:37.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-21T02:02:37.215 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-21T02:02:37.215 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-21T02:02:37.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-21T02:02:37.253 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-21T02:02:37.256 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-21T02:02:37.638 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-21T02:02:37.639 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2024-06-21T02:02:37.639 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-21T02:02:37.698 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-06-21T02:02:37.699 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-21T02:02:37.728 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-21T02:02:37.728 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-21T02:02:37.742 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-21T02:02:37.795 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-21T02:02:37.795 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-21T02:02:37.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-21T02:02:38.218 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-21T02:02:38.228 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-21T02:02:38.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-21T02:02:38.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-21T02:02:38.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-21T02:02:38.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-21T02:02:38.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-21T02:02:38.317 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2024-06-21T02:02:38.342 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-06-21T02:02:38.365 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-21T02:02:38.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-21T02:02:38.410 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-21T02:02:38.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-21T02:02:38.425 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-21T02:02:38.437 INFO:teuthology.task.internal:Transferring archived files... 2024-06-21T02:02:38.438 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi/7765946/remote/smithi092 2024-06-21T02:02:38.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-21T02:02:38.507 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-06-20_17:49:38-orch-squid-release-distro-default-smithi/7765946/remote/smithi173 2024-06-21T02:02:38.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-21T02:02:38.556 INFO:teuthology.task.internal:Removing archive directory... 2024-06-21T02:02:38.556 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-21T02:02:38.559 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-21T02:02:38.603 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-21T02:02:38.613 INFO:teuthology.task.internal:Not uploading archives. 2024-06-21T02:02:38.614 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-21T02:02:38.663 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-21T02:02:38.663 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-21T02:02:38.666 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-21T02:02:38.679 INFO:teuthology.orchestra.run.smithi092.stdout: 265303 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 21 02:02 /home/ubuntu/cephtest 2024-06-21T02:02:38.680 INFO:teuthology.orchestra.run.smithi173.stdout: 265852 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 21 02:02 /home/ubuntu/cephtest 2024-06-21T02:02:38.682 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-21T02:02:38.694 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-21T02:02:38.794 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1641.5481548309326 failure_reason: '"2024-06-21T01:51:42.781372+0000 mon.a (mon.0) 954 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_lflores@folio02 success: false 2024-06-21T02:02:38.794 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-21T02:02:38.893 INFO:teuthology.run:FAIL