2024-04-14T15:10:16.441 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-14T15:10:16.442 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} --name teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101 -- /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101/orig.config.yaml 2024-04-14T15:10:16.470 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-14T15:10:16.558 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101 branch: main description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} email: ceph-qa@ceph.com first_in_suite: false job_id: '7646101' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 workunit: branch: main sha1: 05c9d46e7de5db2cb434c177a6bbfc56f472d09b owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 6509 sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 sleep_before_teardown: 0 subset: 21/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 05c9d46e7de5db2cb434c177a6bbfc56f472d09b targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErCE7+8gl7quRSCjfzY/p94d7HgNCvq0YDZuLdYe2oBhin9jt1ZuEoE37DUQp+Q/4noVNS86OGPA+Ci+98CCe0= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJH71klPmbDk9qxu6LD4ThsEk70DrFl7zWxBL+z/+uJV+gTUIyU1CkXRPtjFDXa4F7Rdc7vmCftFkyueFlT9uzI= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-08_20:08:14 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-14T15:10:16.558 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa; will attempt to use it 2024-04-14T15:10:16.559 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa/tasks 2024-04-14T15:10:16.559 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-14T15:10:16.560 INFO:teuthology.task.internal:Checking packages... 2024-04-14T15:10:16.585 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a971fd5f5719038c70b817bafacd6d5400b1d888' 2024-04-14T15:10:16.585 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-14T15:10:16.585 INFO:teuthology.packaging:ref: None 2024-04-14T15:10:16.585 INFO:teuthology.packaging:tag: None 2024-04-14T15:10:16.585 INFO:teuthology.packaging:branch: main 2024-04-14T15:10:16.585 INFO:teuthology.packaging:sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:10:16.585 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-14T15:10:16.830 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3080.gc2d8c315 2024-04-14T15:10:16.831 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-14T15:10:16.838 INFO:teuthology.task.internal:no buildpackages task found 2024-04-14T15:10:16.838 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-14T15:10:16.861 INFO:teuthology.task.internal:Saving configuration 2024-04-14T15:10:16.871 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-14T15:10:16.878 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T15:10:16.902 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 15:05:17.462356', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErCE7+8gl7quRSCjfzY/p94d7HgNCvq0YDZuLdYe2oBhin9jt1ZuEoE37DUQp+Q/4noVNS86OGPA+Ci+98CCe0='} 2024-04-14T15:10:16.924 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 15:05:17.464368', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJH71klPmbDk9qxu6LD4ThsEk70DrFl7zWxBL+z/+uJV+gTUIyU1CkXRPtjFDXa4F7Rdc7vmCftFkyueFlT9uzI='} 2024-04-14T15:10:16.924 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-14T15:10:16.931 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-04-14T15:10:16.931 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-04-14T15:10:16.931 INFO:teuthology.run_tasks:Running task console_log... 2024-04-14T15:10:17.003 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9abf30b0d0>, signals=[15]) 2024-04-14T15:10:17.004 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-14T15:10:17.010 INFO:teuthology.task.internal:Opening connections... 2024-04-14T15:10:17.010 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-04-14T15:10:17.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:10:17.119 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-04-14T15:10:17.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:10:17.209 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-14T15:10:17.216 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-04-14T15:10:17.235 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-04-14T15:10:17.235 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-04-14T15:10:17.292 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-04-14T15:10:17.292 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-04-14T15:10:17.292 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-04-14T15:10:17.292 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-04-14T15:10:17.292 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-04-14T15:10:17.293 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-04-14T15:10:17.293 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-14T15:10:17.293 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-04-14T15:10:17.293 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-04-14T15:10:17.293 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-14T15:10:17.293 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-04-14T15:10:17.293 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-14T15:10:17.293 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-14T15:10:17.293 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-14T15:10:17.294 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-04-14T15:10:17.325 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-04-14T15:10:17.342 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-04-14T15:10:17.343 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-04-14T15:10:17.400 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-14T15:10:17.401 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-14T15:10:17.401 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-14T15:10:17.401 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-04-14T15:10:17.425 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-14T15:10:17.433 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-14T15:10:17.448 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-14T15:10:17.448 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-04-14T15:10:17.451 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-04-14T15:10:17.467 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-14T15:10:17.473 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-14T15:10:17.473 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-04-14T15:10:17.508 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-04-14T15:10:17.614 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-14T15:10:17.666 INFO:teuthology.run_tasks:Running task kernel... 2024-04-14T15:10:17.681 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-14T15:10:17.681 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-14T15:10:17.681 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-14T15:10:17.682 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-14T15:10:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:10:17.698 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-04-14T15:10:17.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:10:17.699 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-04-14T15:10:17.755 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-361.el9.x86_64 2024-04-14T15:10:17.755 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-361.el9.x86_64 2024-04-14T15:10:17.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-04-14T15:10:17.757 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-361.el9.x86_64 2024-04-14T15:10:17.757 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-361.el9.x86_64 2024-04-14T15:10:17.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-04-14T15:10:19.950 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - AppStream 14 MB/s | 8.1 MB 00:00 2024-04-14T15:10:20.052 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 12 MB/s | 8.1 MB 00:00 2024-04-14T15:10:21.625 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - BaseOS 8.2 MB/s | 2.5 MB 00:00 2024-04-14T15:10:21.802 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.5 MB 00:00 2024-04-14T15:10:22.507 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.0 MB 00:00 2024-04-14T15:10:22.759 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-14T15:10:24.853 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-14T15:10:29.476 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.2 MB 00:00 2024-04-14T15:10:29.942 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 3.0 MB/s | 19 MB 00:06 2024-04-14T15:10:31.160 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-14T15:10:33.305 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:01 2024-04-14T15:10:34.076 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-14T15:10:35.910 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-04-14T15:10:38.578 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-04-14T15:10:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:Modular dependency problems: 2024-04-14T15:10:40.951 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-14T15:10:40.951 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 1: conflicting requests 2024-04-14T15:10:40.951 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.951 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 2: conflicting requests 2024-04-14T15:10:40.951 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.951 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 3: conflicting requests 2024-04-14T15:10:40.951 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.951 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 4: conflicting requests 2024-04-14T15:10:40.951 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.951 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 5: conflicting requests 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 6: conflicting requests 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 7: conflicting requests 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 8: conflicting requests 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 9: conflicting requests 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 10: conflicting requests 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 11: conflicting requests 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.952 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 12: conflicting requests 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 13: conflicting requests 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 14: conflicting requests 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 15: conflicting requests 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 16: conflicting requests 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 17: conflicting requests 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 18: conflicting requests 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.953 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 19: conflicting requests 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 20: conflicting requests 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 21: conflicting requests 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 22: conflicting requests 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 23: conflicting requests 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 24: conflicting requests 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 25: conflicting requests 2024-04-14T15:10:40.954 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 26: conflicting requests 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 27: conflicting requests 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 28: conflicting requests 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 29: conflicting requests 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 30: conflicting requests 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 31: conflicting requests 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 32: conflicting requests 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.955 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 33: conflicting requests 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 34: conflicting requests 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 35: conflicting requests 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 36: conflicting requests 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 37: conflicting requests 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.956 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 39: conflicting requests 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.957 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.958 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:10:40.959 INFO:teuthology.orchestra.run.smithi016.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:10:41.156 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 4.1 MB/s | 21 MB 00:05 2024-04-14T15:10:41.310 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-14T15:10:41.311 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-14T15:10:41.423 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-14T15:10:41.432 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-14T15:10:41.432 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-14T15:10:41.432 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-14T15:10:41.433 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-14T15:10:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-14T15:10:41.433 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-14T15:10:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-14T15:10:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-14T15:10:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-14T15:10:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:10:41.433 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-14T15:10:41.433 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-14T15:10:41.433 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-04-14T15:10:41.433 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:10:41.434 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 77 M 2024-04-14T15:10:41.434 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 123 M 2024-04-14T15:10:41.434 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-14T15:10:41.971 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 769 kB/s | 214 kB 00:00 2024-04-14T15:10:44.129 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 6.2 MB/s | 15 MB 00:02 2024-04-14T15:10:44.596 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): kernel-modules-core-5.14.0-435.el9.x86_6 11 MB/s | 28 MB 00:02 2024-04-14T15:10:44.972 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 10 MB/s | 34 MB 00:03 2024-04-14T15:10:44.972 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:10:44.972 INFO:teuthology.orchestra.run.smithi016.stdout:Total 22 MB/s | 77 MB 00:03 2024-04-14T15:10:45.675 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-14T15:10:46.096 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-14T15:10:46.096 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-14T15:10:46.177 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-14T15:10:47.147 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-14T15:10:47.148 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-14T15:10:48.422 INFO:teuthology.orchestra.run.smithi196.stderr:Modular dependency problems: 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 1: conflicting requests 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 2: conflicting requests 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 3: conflicting requests 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 4: conflicting requests 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 5: conflicting requests 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 6: conflicting requests 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.423 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 7: conflicting requests 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 8: conflicting requests 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 9: conflicting requests 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 10: conflicting requests 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 11: conflicting requests 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 12: conflicting requests 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 13: conflicting requests 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 14: conflicting requests 2024-04-14T15:10:48.424 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 15: conflicting requests 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 16: conflicting requests 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 17: conflicting requests 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 18: conflicting requests 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 19: conflicting requests 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 20: conflicting requests 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 21: conflicting requests 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 22: conflicting requests 2024-04-14T15:10:48.425 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 23: conflicting requests 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 24: conflicting requests 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 25: conflicting requests 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 26: conflicting requests 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 27: conflicting requests 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 28: conflicting requests 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.426 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 29: conflicting requests 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 30: conflicting requests 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 31: conflicting requests 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 32: conflicting requests 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 33: conflicting requests 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 34: conflicting requests 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 35: conflicting requests 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.427 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 36: conflicting requests 2024-04-14T15:10:48.428 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.428 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 37: conflicting requests 2024-04-14T15:10:48.428 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.428 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.428 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.428 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T15:10:48.428 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.428 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.428 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.428 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 39: conflicting requests 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T15:10:48.429 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.430 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T15:10:48.431 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.431 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.431 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.431 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.431 INFO:teuthology.orchestra.run.smithi196.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T15:10:48.431 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.431 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.431 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.431 INFO:teuthology.orchestra.run.smithi196.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T15:10:48.826 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-14T15:10:48.826 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-14T15:10:48.937 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-04-14T15:10:48.947 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-14T15:10:48.947 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-04-14T15:10:48.947 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 77 M 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 123 M 2024-04-14T15:10:48.948 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-04-14T15:10:49.589 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 371 kB/s | 214 kB 00:00 2024-04-14T15:10:49.764 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-14T15:10:50.365 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-14T15:10:50.447 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T15:10:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T15:10:53.534 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T15:10:58.600 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T15:10:58.645 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:11:03.427 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 1.0 MB/s | 15 MB 00:14 2024-04-14T15:11:03.751 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:11:21.332 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-core-5.14.0-435.el9.x86_6 892 kB/s | 28 MB 00:31 2024-04-14T15:11:22.757 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 1.0 MB/s | 34 MB 00:33 2024-04-14T15:11:22.757 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-04-14T15:11:22.758 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.3 MB/s | 77 MB 00:33 2024-04-14T15:11:23.430 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-04-14T15:11:23.807 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-04-14T15:11:23.807 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-04-14T15:11:24.833 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-04-14T15:11:24.833 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-04-14T15:11:27.279 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-04-14T15:11:28.175 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-14T15:11:28.256 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T15:11:31.005 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T15:11:31.151 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T15:11:34.665 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:11:34.665 INFO:teuthology.orchestra.run.smithi016.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-14T15:11:34.665 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:11:34.712 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:11:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:11:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-14T15:11:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T15:11:35.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T15:11:35.479 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:11:35.479 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:11:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-14T15:11:35.479 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-14T15:11:35.479 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-14T15:11:35.479 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-14T15:11:35.480 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-14T15:11:35.480 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:11:35.480 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-14T15:11:35.806 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T15:11:35.873 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T15:11:35.906 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T15:11:35.906 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T15:11:35.907 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-14T15:11:35.907 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-14T15:11:35.907 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-14T15:11:35.907 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-14T15:11:35.907 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-04-14T15:11:35.907 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T15:11:35.991 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T15:11:35.991 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T15:11:35.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-04-14T15:11:36.028 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-04-14T15:11:36.028 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-04-14T15:11:36.028 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.06 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 68.el9 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Group : Unspecified 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 8264254 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : builder@centos.org 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-14T15:11:36.029 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-04-14T15:11:36.030 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-04-14T15:11:36.030 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:11:36.030 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-14T15:11:36.030 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-14T15:11:36.030 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-14T15:11:36.030 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-04-14T15:11:36.030 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:11:36.030 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-04-14T15:11:36.031 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-14T15:11:36.031 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 5.14.0-435.el9.x86_64 2024-04-14T15:11:36.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-14T15:11:36.168 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:11:36.656 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-04-14T15:11:38.082 INFO:teuthology.orchestra.run.smithi016.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-14T15:11:38.121 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-04-14T15:11:38.123 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-04-14T15:11:38.139 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.5shZOqmQm2 2024-04-14T15:11:38.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5shZOqmQm2 2024-04-14T15:11:38.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.5shZOqmQm2 2024-04-14T15:11:38.358 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.5shZOqmQm2 is 6KB 2024-04-14T15:11:38.407 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.5shZOqmQm2 2024-04-14T15:11:38.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-04-14T15:11:38.489 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-14T15:11:38.489 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-14T15:11:38.489 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-14T15:11:38.490 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-14T15:11:38.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-14T15:11:38.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-04-14T15:11:40.944 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:12:08.627 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-14T15:12:08.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-14T15:12:08.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:12:10.110 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:12:10.110 INFO:teuthology.orchestra.run.smithi196.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-14T15:12:10.110 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-14T15:12:10.154 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-14T15:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T15:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T15:12:10.996 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T15:12:10.996 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-14T15:12:10.996 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-04-14T15:12:10.996 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-14T15:12:10.996 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-14T15:12:10.997 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-14T15:12:10.997 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-14T15:12:10.997 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-14T15:12:10.997 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-04-14T15:12:11.362 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T15:12:11.455 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T15:12:11.455 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T15:12:11.455 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-14T15:12:11.455 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-14T15:12:11.455 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-14T15:12:11.455 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-14T15:12:11.455 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-04-14T15:12:11.456 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T15:12:11.536 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T15:12:11.537 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T15:12:11.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-04-14T15:12:11.571 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-04-14T15:12:11.571 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-04-14T15:12:11.571 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-04-14T15:12:11.571 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 68.el9 2024-04-14T15:12:11.571 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-04-14T15:12:11.571 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8264254 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-14T15:12:11.572 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-14T15:12:11.573 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-14T15:12:11.573 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-14T15:12:11.573 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-04-14T15:12:11.573 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-14T15:12:11.573 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-04-14T15:12:11.574 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-14T15:12:11.574 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-435.el9.x86_64 2024-04-14T15:12:11.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-14T15:12:12.155 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-04-14T15:12:13.577 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-14T15:12:13.609 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-04-14T15:12:13.610 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-04-14T15:12:13.630 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.OVMNsx4NQb 2024-04-14T15:12:13.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OVMNsx4NQb 2024-04-14T15:12:13.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.OVMNsx4NQb 2024-04-14T15:12:13.840 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.OVMNsx4NQb is 6KB 2024-04-14T15:12:13.888 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.OVMNsx4NQb 2024-04-14T15:12:13.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-04-14T15:12:13.968 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-14T15:12:13.968 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-14T15:12:13.968 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-14T15:12:13.968 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-14T15:12:13.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-14T15:12:14.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-04-14T15:12:27.167 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-04-14T15:12:36.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-14T15:12:36.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:12:44.130 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-14T15:12:44.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-14T15:12:44.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:12:54.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-04-14T15:13:02.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-04-14T15:13:06.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-14T15:13:06.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:13:09.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-04-14T15:13:11.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-14T15:13:11.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:13:14.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-04-14T15:13:24.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-14T15:13:24.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:13:24.934 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-04-14T15:13:25.308 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-14T15:13:25.308 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-435.el9.x86_64"... 2024-04-14T15:13:25.309 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-04-14T15:13:25.324 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-435.el9.x86_64 2024-04-14T15:13:25.324 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-14T15:13:25.324 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-14T15:13:25.324 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-14T15:13:26.325 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-14T15:13:26.326 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-14T15:13:26.408 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-04-14T15:13:26.454 DEBUG:teuthology.parallel:result is None 2024-04-14T15:13:26.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-14T15:13:26.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:13:44.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-04-14T15:13:59.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-14T15:13:59.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:14:00.455 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-04-14T15:14:00.843 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-14T15:14:00.843 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-435.el9.x86_64"... 2024-04-14T15:14:00.843 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-04-14T15:14:00.857 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-435.el9.x86_64 2024-04-14T15:14:00.857 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-14T15:14:00.858 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-14T15:14:00.858 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-14T15:14:01.859 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-14T15:14:01.859 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-14T15:14:01.948 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-04-14T15:14:01.984 DEBUG:teuthology.parallel:result is None 2024-04-14T15:14:01.985 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-14T15:14:01.995 INFO:teuthology.task.internal:Creating test directory... 2024-04-14T15:14:01.996 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-14T15:14:01.998 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-14T15:14:02.014 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-14T15:14:02.021 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-14T15:14:02.031 INFO:teuthology.task.internal:Creating archive directory... 2024-04-14T15:14:02.032 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-14T15:14:02.057 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-14T15:14:02.097 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-14T15:14:02.104 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-14T15:14:02.104 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-14T15:14:02.123 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-14T15:14:02.161 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T15:14:02.170 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T15:14:02.175 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T15:14:02.183 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T15:14:02.185 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-14T15:14:02.194 INFO:teuthology.task.internal:Configuring sudo... 2024-04-14T15:14:02.194 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-14T15:14:02.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-14T15:14:02.252 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-14T15:14:02.262 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-14T15:14:02.263 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-14T15:14:02.290 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-14T15:14:02.305 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T15:14:02.388 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T15:14:02.438 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T15:14:02.521 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T15:14:02.563 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-14T15:14:02.563 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-14T15:14:02.632 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T15:14:02.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T15:14:02.714 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T15:14:02.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T15:14:02.832 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-14T15:14:02.832 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-14T15:14:02.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-04-14T15:14:02.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-04-14T15:14:02.935 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T15:14:03.019 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T15:14:03.461 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-14T15:14:03.468 INFO:teuthology.task.internal:Starting timer... 2024-04-14T15:14:03.468 INFO:teuthology.run_tasks:Running task pcp... 2024-04-14T15:14:03.568 INFO:teuthology.run_tasks:Running task selinux... 2024-04-14T15:14:03.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-04-14T15:14:03.645 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-04-14T15:14:03.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-04-14T15:14:03.704 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-04-14T15:14:03.706 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-14T15:14:03.707 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-04-14T15:14:03.732 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-04-14T15:14:03.733 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-04-14T15:14:03.764 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-04-14T15:14:03.764 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-04-14T15:14:03.764 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-14T15:14:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:03.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-14T15:14:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:14:03.831 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-14T15:14:03.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-04-14T15:14:03.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-04-14T15:14:03.942 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-14T15:14:03.952 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-14T15:14:03.968 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-14T15:14:03.969 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-14T15:18:23.913 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-04-14T15:18:23.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-14T15:18:23.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:18:24.008 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-04-14T15:18:24.089 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-14T15:18:24.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-14T15:18:24.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T15:18:24.179 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-04-14T15:18:24.257 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-04-14T15:18:24.258 INFO:teuthology.run_tasks:Running task clock... 2024-04-14T15:18:24.269 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-14T15:18:24.269 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-14T15:18:24.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T15:18:24.272 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-14T15:18:24.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T15:18:24.302 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-14T15:18:24.320 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-14T15:18:24.330 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-14T15:18:24.346 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-14T15:18:24.349 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-04-14T15:18:24.364 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-04-14T15:18:24.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-04-14T15:18:24.380 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-14T15:18:24.389 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-04-14T15:18:24.397 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-14T15:18:24.406 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-14T15:18:24.422 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-14T15:18:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-14T15:18:24.459 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T15:18:24.459 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-14T15:18:24.459 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:18:24.459 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:18:24.460 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:18:24.460 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:18:24.476 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-04-14T15:18:24.480 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T15:18:24.480 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-04-14T15:18:24.480 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:18:24.480 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:18:24.480 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:18:24.480 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:18:24.480 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-14T15:18:24.575 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a971fd5f5719038c70b817bafacd6d5400b1d888'} 2024-04-14T15:18:24.575 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-14T15:18:24.576 INFO:tasks.cephadm:Cluster fsid is 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:18:24.576 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-14T15:18:24.576 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.16', 'mon.c': '[v2:172.21.15.16:3301,v1:172.21.15.16:6790]', 'mon.b': '172.21.15.196'} 2024-04-14T15:18:24.576 INFO:tasks.cephadm:First mon is mon.a on smithi016 2024-04-14T15:18:24.576 INFO:tasks.cephadm:First mgr is y 2024-04-14T15:18:24.577 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-14T15:18:24.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-04-14T15:18:24.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname $(hostname -s) 2024-04-14T15:18:24.633 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-14T15:18:24.633 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-14T15:18:24.889 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 14 15:18 /home/ubuntu/cephtest/cephadm 2024-04-14T15:18:24.890 DEBUG:teuthology.orchestra.run.smithi196:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-14T15:18:25.081 INFO:teuthology.orchestra.run.smithi196.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 14 15:18 /home/ubuntu/cephtest/cephadm 2024-04-14T15:18:25.081 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-14T15:18:25.098 DEBUG:teuthology.orchestra.run.smithi196:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-14T15:18:25.122 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-14T15:18:25.123 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-14T15:18:25.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-14T15:18:26.000 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-14T15:18:26.082 INFO:teuthology.orchestra.run.smithi196.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-14T15:18:49.382 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-04-14T15:18:49.382 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-14T15:18:49.382 INFO:teuthology.orchestra.run.smithi196.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-14T15:18:49.382 INFO:teuthology.orchestra.run.smithi196.stdout: "repo_digests": [ 2024-04-14T15:18:49.383 INFO:teuthology.orchestra.run.smithi196.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-14T15:18:49.383 INFO:teuthology.orchestra.run.smithi196.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-14T15:18:49.383 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-04-14T15:18:49.383 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-04-14T15:18:49.926 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:18:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-14T15:18:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-14T15:18:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-04-14T15:18:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-14T15:18:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-14T15:18:49.927 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-14T15:18:49.927 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:18:49.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-04-14T15:18:49.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph 2024-04-14T15:18:50.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-04-14T15:18:50.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /etc/ceph 2024-04-14T15:18:50.104 INFO:tasks.cephadm:Writing seed config... 2024-04-14T15:18:50.105 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-04-14T15:18:50.105 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-14T15:18:50.105 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-14T15:18:50.105 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-04-14T15:18:50.105 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-14T15:18:50.105 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-14T15:18:50.105 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-14T15:18:50.106 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-14T15:18:50.106 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-14T15:18:50.106 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-14T15:18:50.106 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-14T15:18:50.121 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3ca5917a-fa72-11ee-bc8e-c7b262605968 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-14T15:18:50.121 DEBUG:teuthology.orchestra.run.smithi016:mon.a> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.a.service 2024-04-14T15:18:50.163 DEBUG:teuthology.orchestra.run.smithi016:mgr.y> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.y.service 2024-04-14T15:18:50.206 INFO:tasks.cephadm:Bootstrapping... 2024-04-14T15:18:50.206 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:18:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:-------------------------------------------------------------------------------- 2024-04-14T15:18:50.407 INFO:teuthology.orchestra.run.smithi016.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '3ca5917a-fa72-11ee-bc8e-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.16', '--skip-admin-label'] 2024-04-14T15:18:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: 4.9.4-dev 2024-04-14T15:18:50.433 INFO:teuthology.orchestra.run.smithi016.stderr:Verifying podman|docker is present... 2024-04-14T15:18:50.453 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: 4.9.4-dev 2024-04-14T15:18:50.455 INFO:teuthology.orchestra.run.smithi016.stderr:Verifying lvm2 is present... 2024-04-14T15:18:50.456 INFO:teuthology.orchestra.run.smithi016.stderr:Verifying time synchronization is in place... 2024-04-14T15:18:50.462 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-14T15:18:50.468 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: inactive 2024-04-14T15:18:50.475 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: enabled 2024-04-14T15:18:50.480 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: active 2024-04-14T15:18:50.481 INFO:teuthology.orchestra.run.smithi016.stderr:Unit chronyd.service is enabled and running 2024-04-14T15:18:50.481 INFO:teuthology.orchestra.run.smithi016.stderr:Repeating the final host check... 2024-04-14T15:18:50.501 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: 4.9.4-dev 2024-04-14T15:18:50.503 INFO:teuthology.orchestra.run.smithi016.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-14T15:18:50.503 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl is present 2024-04-14T15:18:50.503 INFO:teuthology.orchestra.run.smithi016.stderr:lvcreate is present 2024-04-14T15:18:50.509 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-14T15:18:50.515 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: inactive 2024-04-14T15:18:50.521 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: enabled 2024-04-14T15:18:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: active 2024-04-14T15:18:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:Unit chronyd.service is enabled and running 2024-04-14T15:18:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:Host looks OK 2024-04-14T15:18:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:Cluster fsid: 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:18:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:Acquiring lock 140597930098016 on /run/cephadm/3ca5917a-fa72-11ee-bc8e-c7b262605968.lock 2024-04-14T15:18:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:Lock 140597930098016 acquired on /run/cephadm/3ca5917a-fa72-11ee-bc8e-c7b262605968.lock 2024-04-14T15:18:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:Verifying IP 172.21.15.16 port 3300 ... 2024-04-14T15:18:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:Verifying IP 172.21.15.16 port 6789 ... 2024-04-14T15:18:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:Base mon IP is 172.21.15.16, final addrv is [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-04-14T15:18:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.16 metric 100 2024-04-14T15:18:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-04-14T15:18:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-14T15:18:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-14T15:18:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-14T15:18:50.537 INFO:teuthology.orchestra.run.smithi016.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-14T15:18:50.537 INFO:teuthology.orchestra.run.smithi016.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-14T15:18:50.537 INFO:teuthology.orchestra.run.smithi016.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-14T15:18:50.537 INFO:teuthology.orchestra.run.smithi016.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-14T15:18:50.537 INFO:teuthology.orchestra.run.smithi016.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-14T15:18:50.537 INFO:teuthology.orchestra.run.smithi016.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-14T15:18:50.538 INFO:teuthology.orchestra.run.smithi016.stderr:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-04-14T15:18:50.538 INFO:teuthology.orchestra.run.smithi016.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-14T15:18:50.539 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-14T15:18:50.643 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-14T15:18:51.199 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: Getting image source signatures 2024-04-14T15:18:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-14T15:18:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-14T15:18:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-14T15:18:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-14T15:18:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-14T15:18:51.200 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-14T15:18:51.322 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: Writing manifest to image destination 2024-04-14T15:18:51.441 INFO:teuthology.orchestra.run.smithi016.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-14T15:18:51.832 INFO:teuthology.orchestra.run.smithi016.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-14T15:18:52.391 INFO:teuthology.orchestra.run.smithi016.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-14T15:18:52.391 INFO:teuthology.orchestra.run.smithi016.stderr:Extracting ceph user uid/gid from container image... 2024-04-14T15:18:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:stat: 167 167 2024-04-14T15:18:53.180 INFO:teuthology.orchestra.run.smithi016.stderr:Creating initial keys... 2024-04-14T15:18:53.596 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph-authtool: AQDd8xtmbMiRHhAAgfVusuIT3sitK3fAhbDDpg== 2024-04-14T15:18:54.576 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph-authtool: AQDe8xtmSJfYFRAAxrQ836GzcYpu3Qsk5lAfPA== 2024-04-14T15:18:55.582 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph-authtool: AQDf8xtm3zS1HRAADqdWGqzOIMxn0N9xNsg6vQ== 2024-04-14T15:18:56.032 INFO:teuthology.orchestra.run.smithi016.stderr:Creating initial monmap... 2024-04-14T15:18:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-14T15:18:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-14T15:18:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:18:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-14T15:18:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:monmaptool for a [v2:172.21.15.16:3300,v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-14T15:18:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:setting min_mon_release = octopus 2024-04-14T15:18:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/monmaptool: set fsid to 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:18:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-14T15:18:56.986 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-14T15:18:56.986 INFO:teuthology.orchestra.run.smithi016.stderr:Creating mon... 2024-04-14T15:18:57.917 INFO:teuthology.orchestra.run.smithi016.stderr:create mon.a on 2024-04-14T15:18:58.157 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-14T15:18:58.304 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968.target → /etc/systemd/system/ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968.target. 2024-04-14T15:18:58.304 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968.target → /etc/systemd/system/ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968.target. 2024-04-14T15:18:58.600 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: Failed to reset failed state of unit ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.a.service: Unit ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.a.service not loaded. 2024-04-14T15:18:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968.target.wants/ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.a.service → /etc/systemd/system/ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@.service. 2024-04-14T15:18:59.269 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:18:59 smithi016 ceph-mon[24517]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-14T15:18:59.269 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:18:59 smithi016 bash[24498]: 641cc2bab0971cf39925a8fdef7bc16e46c30aae1836aadddd23a45240d878d6 2024-04-14T15:18:59.281 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: disabled 2024-04-14T15:18:59.288 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: inactive 2024-04-14T15:18:59.288 INFO:teuthology.orchestra.run.smithi016.stderr:firewalld.service is not enabled 2024-04-14T15:18:59.289 INFO:teuthology.orchestra.run.smithi016.stderr:Not possible to enable service . firewalld.service is not available 2024-04-14T15:18:59.289 INFO:teuthology.orchestra.run.smithi016.stderr:Waiting for mon to start... 2024-04-14T15:18:59.289 INFO:teuthology.orchestra.run.smithi016.stderr:Waiting for mon... 2024-04-14T15:18:59.530 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:18:59 smithi016 systemd[1]: Started Ceph mon.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:18:59.816 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: cluster: 2024-04-14T15:18:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: id: 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:18:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-14T15:18:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 2024-04-14T15:18:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: services: 2024-04-14T15:18:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.589995s) 2024-04-14T15:18:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-14T15:18:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-14T15:18:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 2024-04-14T15:18:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: data: 2024-04-14T15:18:59.817 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-14T15:18:59.818 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-14T15:18:59.818 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-14T15:18:59.818 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: pgs: 2024-04-14T15:18:59.818 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 2024-04-14T15:19:00.345 INFO:teuthology.orchestra.run.smithi016.stderr:mon is available 2024-04-14T15:19:00.345 INFO:teuthology.orchestra.run.smithi016.stderr:Assimilating anything we can from ceph.conf... 2024-04-14T15:19:00.468 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:00 smithi016 ceph-mon[24517]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-14T15:19:00.468 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:00 smithi016 ceph-mon[24517]: monmap e1: 1 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} 2024-04-14T15:19:00.468 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:00 smithi016 ceph-mon[24517]: fsmap 2024-04-14T15:19:00.468 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:00 smithi016 ceph-mon[24517]: osdmap e1: 0 total, 0 up, 0 in 2024-04-14T15:19:00.468 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:00 smithi016 ceph-mon[24517]: mgrmap e1: no daemons active 2024-04-14T15:19:00.468 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:00 smithi016 ceph-mon[24517]: from='client.? 172.21.15.16:0/1362727343' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T15:19:00.860 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 2024-04-14T15:19:00.860 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: [global] 2024-04-14T15:19:00.860 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: fsid = 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:19:00.860 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-04-14T15:19:00.860 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-14T15:19:00.860 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-14T15:19:00.860 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-14T15:19:00.860 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-14T15:19:00.860 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 2024-04-14T15:19:00.861 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: [mgr] 2024-04-14T15:19:00.861 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-04-14T15:19:00.861 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-14T15:19:00.861 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 2024-04-14T15:19:00.861 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: [osd] 2024-04-14T15:19:00.861 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-14T15:19:00.861 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-14T15:19:01.377 INFO:teuthology.orchestra.run.smithi016.stderr:Generating new minimal ceph.conf... 2024-04-14T15:19:01.483 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:01 smithi016 ceph-mon[24517]: from='client.? 172.21.15.16:0/47750983' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-14T15:19:01.483 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:01 smithi016 ceph-mon[24517]: from='client.? 172.21.15.16:0/47750983' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-14T15:19:02.343 INFO:teuthology.orchestra.run.smithi016.stderr:Restarting the monitor... 2024-04-14T15:19:02.523 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:02 smithi016 ceph-mon[24517]: from='client.? 172.21.15.16:0/777671436' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:19:02.523 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:02 smithi016 systemd[1]: Stopping Ceph mon.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:19:02.523 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:02 smithi016 bash[24746]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon.a" found: no such container 2024-04-14T15:19:02.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a[24513]: 2024-04-14T15:19:02.521+0000 7fa6bb706700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:19:02.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a[24513]: 2024-04-14T15:19:02.521+0000 7fa6bb706700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-14T15:19:02.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:02 smithi016 podman[24757]: 2024-04-14 15:19:02.665745523 +0000 UTC m=+0.270785893 container died 641cc2bab0971cf39925a8fdef7bc16e46c30aae1836aadddd23a45240d878d6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, RELEASE=HEAD, release=754, GIT_CLEAN=True, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc.) 2024-04-14T15:19:03.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:02 smithi016 podman[24757]: 2024-04-14 15:19:02.950420959 +0000 UTC m=+0.555461325 container cleanup 641cc2bab0971cf39925a8fdef7bc16e46c30aae1836aadddd23a45240d878d6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.buildah.version=1.19.8, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_BRANCH=HEAD, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, name=centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-14T15:19:03.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:02 smithi016 bash[24757]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a 2024-04-14T15:19:03.192 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:02 smithi016 bash[24785]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon.a" found: no such container 2024-04-14T15:19:03.492 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 podman[24771]: 2024-04-14 15:19:03.192947925 +0000 UTC m=+0.530539478 container remove 641cc2bab0971cf39925a8fdef7bc16e46c30aae1836aadddd23a45240d878d6 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , distribution-scope=public, release=754, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, architecture=x86_64, version=8, RELEASE=HEAD, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-14T15:19:03.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-14T15:19:03.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 systemd[1]: Stopped Ceph mon.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:19:03.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 systemd[1]: Starting Ceph mon.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:19:03.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 podman[24851]: 2024-04-14 15:19:03.401911525 +0000 UTC m=+0.085072243 container create e8cfd2da2fd965d6d8e193f500806c654d6a7c39f077d54f30dea8298fdf5308 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, version=8, ceph=True, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_CLEAN=True, name=centos-stream, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD) 2024-04-14T15:19:03.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 podman[24851]: 2024-04-14 15:19:03.335878749 +0000 UTC m=+0.019039486 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-14T15:19:03.657 INFO:teuthology.orchestra.run.smithi016.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-14T15:19:03.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 podman[24851]: 2024-04-14 15:19:03.571656981 +0000 UTC m=+0.254817699 container init e8cfd2da2fd965d6d8e193f500806c654d6a7c39f077d54f30dea8298fdf5308 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, distribution-scope=public, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, name=centos-stream, ceph=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_BRANCH=HEAD) 2024-04-14T15:19:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 podman[24851]: 2024-04-14 15:19:03.574695894 +0000 UTC m=+0.257856613 container start e8cfd2da2fd965d6d8e193f500806c654d6a7c39f077d54f30dea8298fdf5308 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, io.openshift.expose-services=, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, GIT_BRANCH=HEAD, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-14T15:19:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T15:19:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-14T15:19:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: pidfile_write: ignore empty --pid-file 2024-04-14T15:19:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: load: jerasure load: lrc 2024-04-14T15:19:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: RocksDB version: 6.15.5 2024-04-14T15:19:03.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-14T15:19:03.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Compile date Apr 18 2022 2024-04-14T15:19:03.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: DB SUMMARY 2024-04-14T15:19:03.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: DB Session ID: I5ZCN5MMSILAMY8TUHKS 2024-04-14T15:19:03.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: CURRENT file: CURRENT 2024-04-14T15:19:03.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T15:19:03.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-14T15:19:03.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-14T15:19:03.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82617 ; 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.error_if_exists: 0 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.create_if_missing: 0 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.paranoid_checks: 1 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.env: 0x55ff146c6860 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.fs: Posix File System 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.info_log: 0x55ff154c3dc0 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.statistics: (nil) 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.use_fsync: 0 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_log_file_size: 0 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T15:19:03.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.allow_fallocate: 1 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.use_direct_reads: 0 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.db_log_dir: 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T15:19:03.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.write_buffer_manager: 0x55ff155b42a0 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.unordered_write: 0 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T15:19:03.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.row_cache: None 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.wal_filter: None 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.preserve_deletes: 0 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.two_write_queues: 0 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.atomic_flush: 0 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.log_readahead_size: 0 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T15:19:03.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_background_jobs: 2 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_background_compactions: -1 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_subcompactions: 1 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_open_files: -1 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T15:19:03.867 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_background_flushes: -1 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Compression algorithms supported: 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: kZSTD supported: 0 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: kXpressCompression supported: 0 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: kLZ4Compression supported: 1 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: kBZip2Compression supported: 0 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: kZlibCompression supported: 1 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: kSnappyCompression supported: 1 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-04-14T15:19:03.868 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.merge_operator: 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_filter: None 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ff1548fd00) 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T15:19:03.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: hash_index_allow_collision: 1 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 1 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x55ff154fa170 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-04-14T15:19:03.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 4 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compression: NoCompression 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T15:19:03.871 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.num_levels: 7 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T15:19:03.872 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T15:19:03.873 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:19:03.873 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T15:19:03.873 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T15:19:03.873 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T15:19:03.874 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.arena_block_size: 4194304 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T15:19:03.875 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.table_properties_collectors: 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.inplace_update_support: 0 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.bloom_locality: 0 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.max_successive_merges: 0 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.ttl: 2592000 2024-04-14T15:19:03.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.enable_blob_files: false 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.min_blob_size: 0 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107943600465, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107943601182, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79167, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77430, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71739, "raw_average_value_size": 339, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713107943, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "1a531674-f9b5-4d47-abbd-9b5d3b09e6c8", "db_session_id": "I5ZCN5MMSILAMY8TUHKS"}} 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-14T15:19:03.877 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713107943601523, "job": 1, "event": "recovery_finished"} 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55ff154e0a80 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: DB pointer 0x55ff154f0000 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:19:03.878 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 2/0 79.07 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.1 0.00 0.00 1 0.000 0 0 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 79.07 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.1 0.00 0.00 1 0.000 0 0 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.1 0.00 0.00 1 0.000 0 0 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.1 0.00 0.00 1 0.000 0 0 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:19:03.879 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 25.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 25.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 2/0 79.07 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.1 0.00 0.00 1 0.000 0 0 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 79.07 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.1 0.00 0.00 1 0.000 0 0 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:19:03.880 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.1 0.00 0.00 1 0.000 0 0 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 25.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: starting mon.a rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: mon.a@-1(???) e1 preinit fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:19:03.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: mon.a@-1(???).mds e1 new map 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: mon.a@-1(???).mds e1 print_map 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta expand map: {default=false} 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta from 'false' to 'false' 2024-04-14T15:19:03.882 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta expanded map: {default=false} 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta expand map: {default=info} 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta from 'info' to 'info' 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta expanded map: {default=info} 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta expand map: {default=daemon} 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta expanded map: {default=daemon} 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta expand map: {default=debug} 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta from 'debug' to 'debug' 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: expand_channel_meta expanded map: {default=debug} 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 bash[24851]: e8cfd2da2fd965d6d8e193f500806c654d6a7c39f077d54f30dea8298fdf5308 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 systemd[1]: Started Ceph mon.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: monmap e1: 1 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: fsmap 2024-04-14T15:19:03.883 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: osdmap e1: 0 total, 0 up, 0 in 2024-04-14T15:19:03.884 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:03 smithi016 ceph-mon[24869]: mgrmap e1: no daemons active 2024-04-14T15:19:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-14T15:19:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:19:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:Creating mgr... 2024-04-14T15:19:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:Verifying port 9283 ... 2024-04-14T15:19:04.800 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: Failed to reset failed state of unit ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.y.service: Unit ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.y.service not loaded. 2024-04-14T15:19:04.807 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968.target.wants/ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@.service. 2024-04-14T15:19:05.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:04 smithi016 systemd[1]: Starting Ceph mgr.y for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:19:05.489 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:05 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/16094027' entity='client.admin' 2024-04-14T15:19:05.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:05 smithi016 podman[25084]: 2024-04-14 15:19:05.089464239 +0000 UTC m=+0.017811692 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-14T15:19:05.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:05 smithi016 podman[25084]: 2024-04-14 15:19:05.195326203 +0000 UTC m=+0.123673653 container create 8678f197809c666b732a6b6920caf13da144ca0f1ac833279896903d065f8fd5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y, RELEASE=HEAD, ceph=True, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, com.redhat.component=centos-stream-container, release=754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.openshift.expose-services=, name=centos-stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True) 2024-04-14T15:19:05.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:05 smithi016 podman[25084]: 2024-04-14 15:19:05.389370178 +0000 UTC m=+0.317717620 container init 8678f197809c666b732a6b6920caf13da144ca0f1ac833279896903d065f8fd5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, ceph=True, architecture=x86_64, name=centos-stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., distribution-scope=public, version=8, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-14T15:19:05.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:05 smithi016 podman[25084]: 2024-04-14 15:19:05.391784001 +0000 UTC m=+0.320131444 container start 8678f197809c666b732a6b6920caf13da144ca0f1ac833279896903d065f8fd5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , name=centos-stream, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, vcs-type=git, io.openshift.expose-services=, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container) 2024-04-14T15:19:05.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:05 smithi016 bash[25084]: 8678f197809c666b732a6b6920caf13da144ca0f1ac833279896903d065f8fd5 2024-04-14T15:19:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: disabled 2024-04-14T15:19:05.510 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: inactive 2024-04-14T15:19:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:firewalld.service is not enabled 2024-04-14T15:19:05.511 INFO:teuthology.orchestra.run.smithi016.stderr:Not possible to enable service . firewalld.service is not available 2024-04-14T15:19:05.519 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: disabled 2024-04-14T15:19:05.527 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: inactive 2024-04-14T15:19:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:firewalld.service is not enabled 2024-04-14T15:19:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-14T15:19:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:Waiting for mgr to start... 2024-04-14T15:19:05.528 INFO:teuthology.orchestra.run.smithi016.stderr:Waiting for mgr... 2024-04-14T15:19:05.760 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:05 smithi016 systemd[1]: Started Ceph mgr.y for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:19:06.030 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:05 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:05.897+0000 7f1e04674000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:19:06.030 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:05 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:05.990+0000 7f1e04674000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:19:06.033 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 2024-04-14T15:19:06.033 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: { 2024-04-14T15:19:06.033 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "fsid": "3ca5917a-fa72-11ee-bc8e-c7b262605968", 2024-04-14T15:19:06.033 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "health": { 2024-04-14T15:19:06.033 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-14T15:19:06.033 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "checks": {}, 2024-04-14T15:19:06.033 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "mutes": [] 2024-04-14T15:19:06.033 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:06.033 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "quorum": [ 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 0 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: ], 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "a" 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: ], 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "monmap": { 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "osdmap": { 2024-04-14T15:19:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T15:19:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-14T15:19:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-14T15:19:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-14T15:19:06.035 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-14T15:19:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-14T15:19:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-14T15:19:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "pgmap": { 2024-04-14T15:19:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "fsmap": { 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "available": false, 2024-04-14T15:19:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "modules": [ 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "iostat", 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "nfs", 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "restful" 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: ], 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "services": {} 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "servicemap": { 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "modified": "2024-04-14T15:18:59.224924+0000", 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "services": {} 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-14T15:19:06.040 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: } 2024-04-14T15:19:06.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:06 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3225196975' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T15:19:06.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:06 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:06.607+0000 7f1e04674000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:19:06.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:06 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:06.696+0000 7f1e04674000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:19:06.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:06 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:06.781+0000 7f1e04674000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:19:07.249 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:06 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:06.931+0000 7f1e04674000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:19:07.249 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:07.013+0000 7f1e04674000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:19:07.249 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:07.124+0000 7f1e04674000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:19:07.249 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:07.208+0000 7f1e04674000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:19:07.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:07.372+0000 7f1e04674000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:19:07.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:07.458+0000 7f1e04674000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:19:08.835 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:08.513+0000 7f1e04674000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:19:08.835 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:08.644+0000 7f1e04674000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:19:08.835 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:08.835+0000 7f1e04674000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:19:09.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:08.922+0000 7f1e04674000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:19:09.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:09.059+0000 7f1e04674000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:19:09.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:09.379+0000 7f1e04674000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:19:09.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:09.601+0000 7f1e04674000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:19:09.687 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:09.686+0000 7f1e04674000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:19:10.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:09.859+0000 7f1e04674000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:19:11.093 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:10 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:10.794+0000 7f1e04674000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:19:11.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:11.092+0000 7f1e04674000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:19:11.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:11.187+0000 7f1e04674000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:19:11.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:11.341+0000 7f1e04674000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:19:11.628 INFO:teuthology.orchestra.run.smithi016.stderr:mgr not available, waiting (1/15)... 2024-04-14T15:19:11.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: Activating manager daemon y 2024-04-14T15:19:11.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: mgrmap e2: y(active, starting, since 0.0028212s) 2024-04-14T15:19:11.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:19:11.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:19:11.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:19:11.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:19:11.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:19:11.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: Manager daemon y is now available 2024-04-14T15:19:11.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:19:11.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:19:11.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' 2024-04-14T15:19:11.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' 2024-04-14T15:19:11.831 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:11 smithi016 ceph-mon[24869]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' 2024-04-14T15:19:13.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:13 smithi016 ceph-mon[24869]: mgrmap e3: y(active, since 1.00716s) 2024-04-14T15:19:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 2024-04-14T15:19:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: { 2024-04-14T15:19:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "fsid": "3ca5917a-fa72-11ee-bc8e-c7b262605968", 2024-04-14T15:19:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "health": { 2024-04-14T15:19:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-14T15:19:14.324 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "checks": {}, 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "mutes": [] 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "quorum": [ 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 0 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: ], 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "a" 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: ], 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "monmap": { 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:14.325 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "osdmap": { 2024-04-14T15:19:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T15:19:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-14T15:19:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-14T15:19:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-14T15:19:14.326 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-14T15:19:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-14T15:19:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-14T15:19:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "pgmap": { 2024-04-14T15:19:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-14T15:19:14.329 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "fsmap": { 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "available": true, 2024-04-14T15:19:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "modules": [ 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "iostat", 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "nfs", 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "restful" 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: ], 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "services": {} 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "servicemap": { 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "modified": "2024-04-14T15:18:59.224924+0000", 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "services": {} 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: }, 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-14T15:19:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: } 2024-04-14T15:19:14.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:14 smithi016 ceph-mon[24869]: mgrmap e4: y(active, since 2s) 2024-04-14T15:19:14.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:14 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4259571193' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T15:19:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:mgr is available 2024-04-14T15:19:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 2024-04-14T15:19:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: [global] 2024-04-14T15:19:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: fsid = 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:19:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-14T15:19:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-14T15:19:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-14T15:19:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-14T15:19:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 2024-04-14T15:19:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: [mgr] 2024-04-14T15:19:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-14T15:19:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 2024-04-14T15:19:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: [osd] 2024-04-14T15:19:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-14T15:19:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-14T15:19:15.771 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:15 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4275195754' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-14T15:19:15.771 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:15 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4275195754' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-14T15:19:15.985 INFO:teuthology.orchestra.run.smithi016.stderr:Enabling cephadm module... 2024-04-14T15:19:16.997 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:16 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1823767811' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-14T15:19:16.997 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:16 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ignoring --setuser ceph since I am not root 2024-04-14T15:19:16.998 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:16 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ignoring --setgroup ceph since I am not root 2024-04-14T15:19:17.286 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:17 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:17.187+0000 7fd561d8f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:19:17.559 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:17 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:17.286+0000 7fd561d8f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:19:17.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:17 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1823767811' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-14T15:19:17.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:17 smithi016 ceph-mon[24869]: mgrmap e5: y(active, since 5s) 2024-04-14T15:19:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: { 2024-04-14T15:19:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-14T15:19:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "available": true, 2024-04-14T15:19:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-14T15:19:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-14T15:19:17.896 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: } 2024-04-14T15:19:18.305 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:17 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:17.906+0000 7fd561d8f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:19:18.305 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:18 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:18.000+0000 7fd561d8f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:19:18.306 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:18 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:18.085+0000 7fd561d8f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:19:18.306 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:18 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:18.236+0000 7fd561d8f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:19:18.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:18 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:18.319+0000 7fd561d8f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:19:18.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:18 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:18.430+0000 7fd561d8f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:19:18.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:18 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:18.516+0000 7fd561d8f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:19:19.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:18 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1653464846' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-14T15:19:19.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:18 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:18.682+0000 7fd561d8f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:19:19.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:18 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:18.770+0000 7fd561d8f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:19:20.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:19 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:19.842+0000 7fd561d8f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:19:20.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:19 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:19.978+0000 7fd561d8f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:19:20.560 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:20 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:20.175+0000 7fd561d8f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:19:20.560 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:20 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:20.265+0000 7fd561d8f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:19:20.560 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:20 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:20.406+0000 7fd561d8f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:19:20.851 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:20 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:20.729+0000 7fd561d8f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:19:20.854 INFO:teuthology.orchestra.run.smithi016.stderr:Waiting for the mgr to restart... 2024-04-14T15:19:20.854 INFO:teuthology.orchestra.run.smithi016.stderr:Waiting for mgr epoch 5... 2024-04-14T15:19:21.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:20 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:20.965+0000 7fd561d8f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:19:21.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:21 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:21.052+0000 7fd561d8f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:19:21.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:21 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:21.230+0000 7fd561d8f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:19:22.482 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:22.173+0000 7fd561d8f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:19:22.482 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:22.481+0000 7fd561d8f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:19:22.734 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:22.577+0000 7fd561d8f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:19:22.734 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:22.733+0000 7fd561d8f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:19:23.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: Active manager daemon y restarted 2024-04-14T15:19:23.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: Activating manager daemon y 2024-04-14T15:19:23.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: osdmap e2: 0 total, 0 up, 0 in 2024-04-14T15:19:23.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: mgrmap e6: y(active, starting, since 0.0522864s) 2024-04-14T15:19:23.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:19:23.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:19:23.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:19:23.798 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:19:23.798 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:19:23.798 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: Manager daemon y is now available 2024-04-14T15:19:23.798 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:23.798 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:23.798 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:23.798 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:23.798 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:23.799 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:19:23.799 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:23 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:19:23.799 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: { 2024-04-14T15:19:23.800 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-14T15:19:23.800 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "initialized": true 2024-04-14T15:19:23.800 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: } 2024-04-14T15:19:24.369 INFO:teuthology.orchestra.run.smithi016.stderr:mgr epoch 5 is available 2024-04-14T15:19:24.370 INFO:teuthology.orchestra.run.smithi016.stderr:Setting orchestrator backend to cephadm... 2024-04-14T15:19:24.972 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:24 smithi016 ceph-mon[24869]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-14T15:19:24.972 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:24 smithi016 ceph-mon[24869]: mgrmap e7: y(active, since 1.05476s) 2024-04-14T15:19:24.972 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:24 smithi016 ceph-mon[24869]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-14T15:19:25.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:25 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:19:25] ENGINE Bus STARTING 2024-04-14T15:19:25.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:25 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:19:25] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:19:25.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:25 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:19:25] ENGINE Bus STARTED 2024-04-14T15:19:25.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:25 smithi016 ceph-mon[24869]: mgrmap e8: y(active, since 2s) 2024-04-14T15:19:25.981 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:25 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:25.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:25 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:25.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:25 smithi016 ceph-mon[24869]: [14/Apr/2024:15:19:25] ENGINE Bus STARTING 2024-04-14T15:19:25.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:25 smithi016 ceph-mon[24869]: [14/Apr/2024:15:19:25] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:19:25.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:25 smithi016 ceph-mon[24869]: [14/Apr/2024:15:19:25] ENGINE Bus STARTED 2024-04-14T15:19:25.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:25 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:25.982 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:25 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:26.370 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: value unchanged 2024-04-14T15:19:26.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:26 smithi016 ceph-mon[24869]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:19:26.957 INFO:teuthology.orchestra.run.smithi016.stderr:Generating ssh key... 2024-04-14T15:19:28.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-mon[24869]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:19:28.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-mon[24869]: Generating ssh key... 2024-04-14T15:19:28.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: Generating public/private rsa key pair. 2024-04-14T15:19:28.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: Your identification has been saved in /tmp/tmptjsbrkct/key. 2024-04-14T15:19:28.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: Your public key has been saved in /tmp/tmptjsbrkct/key.pub. 2024-04-14T15:19:28.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: The key fingerprint is: 2024-04-14T15:19:28.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: SHA256:Wnuz5sdssMG+Wb7N63JonD9ZJESkMK6lRqYcKd2pAFQ ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:19:28.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: The key's randomart image is: 2024-04-14T15:19:28.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: +---[RSA 3072]----+ 2024-04-14T15:19:28.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: | .o.E o oo | 2024-04-14T15:19:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: | . . o o o .. | 2024-04-14T15:19:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: | o + = o .. | 2024-04-14T15:19:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: | + * + . .| 2024-04-14T15:19:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: | + S. o | 2024-04-14T15:19:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: | + .+ .| 2024-04-14T15:19:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: | . ..oB.o o | 2024-04-14T15:19:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: | .+*@o+ | 2024-04-14T15:19:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: | o==o**o | 2024-04-14T15:19:28.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: +----[SHA256]-----+ 2024-04-14T15:19:28.998 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:28 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:28.998 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:28 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:28.999 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoHt4gx1d74c/02rtMNGRzV0IwgUZdf+nTGLujNob0s1SLXS9laSwZe9UJ1CKzdb3l8bAOlOfig/Qbh3r9rHuU3vQazmL9J7Omifu7UrxX1UsdDvqiRZDrOfGD8K6yfWon0n8cfFtN7cL1YWejFRfOTnhLP1i9qbhDTmUFuH7IbwQubP2ujA+OawgOYKbX5Hd5Le71Rl8S0HzOG6zkfWWM8JuhgE8oTmmZfydfXTvQ2ShGv/2kC2tNDiO18Lh0W1fp8yk6T0Xrbzqx4owsDZ4JN0vk/pqwTExvYt55nsSWx4U6AVs4R+303sXorc6jJZdXax7mbcUhd48GlAvB3rC7lut9H5e1IJQlMF8M2xHKxnDDD3v0uzTmPiA2zjfKOQGFeDg5F259k+c2lnHoBF8LjAh7HBayjGeCa8JUysqzbwVnkx5YAPXDLOmj8A/qmJsI0GFxirBbxURH6LbS6SqGZLbxBEh3BFqjfgIqiXHz+zDBbGiukoce8/8v3iEt9T0= ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:19:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-14T15:19:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:Adding key to root@localhost authorized_keys... 2024-04-14T15:19:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:Adding host smithi016... 2024-04-14T15:19:29.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:29 smithi016 ceph-mon[24869]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:19:30.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:30 smithi016 ceph-mon[24869]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:19:30.821 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:30 smithi016 ceph-mon[24869]: Deploying cephadm binary to smithi016 2024-04-14T15:19:31.258 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: Added host 'smithi016' with addr '172.21.15.16' 2024-04-14T15:19:31.861 INFO:teuthology.orchestra.run.smithi016.stderr:Deploying unmanaged mon service... 2024-04-14T15:19:32.513 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:32 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:32.514 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:32 smithi016 ceph-mon[24869]: Added host smithi016 2024-04-14T15:19:32.514 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:32 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:32.514 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-14T15:19:33.253 INFO:teuthology.orchestra.run.smithi016.stderr:Deploying unmanaged mgr service... 2024-04-14T15:19:33.838 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:33 smithi016 ceph-mon[24869]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:19:33.838 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:33 smithi016 ceph-mon[24869]: Saving service mon spec with placement count:5 2024-04-14T15:19:33.838 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:33 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:33.938 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-14T15:19:35.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:34 smithi016 ceph-mon[24869]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:19:35.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:34 smithi016 ceph-mon[24869]: Saving service mgr spec with placement count:2 2024-04-14T15:19:35.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:34 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:35.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:34 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:36.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:36 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:36.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:36 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2550926075' entity='client.admin' 2024-04-14T15:19:38.467 INFO:teuthology.orchestra.run.smithi016.stderr:Enabling the dashboard module... 2024-04-14T15:19:38.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:38 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:38.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:38 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3102626295' entity='client.admin' 2024-04-14T15:19:38.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:38 smithi016 ceph-mon[24869]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:19:39.991 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:39 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4151726119' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-14T15:19:39.991 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:39 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ignoring --setuser ceph since I am not root 2024-04-14T15:19:39.992 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:39 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ignoring --setgroup ceph since I am not root 2024-04-14T15:19:40.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:40 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:40.126+0000 7fbd44b7c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:19:40.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:40 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:40.218+0000 7fbd44b7c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:19:40.703 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:40 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4151726119' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-14T15:19:40.703 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:40 smithi016 ceph-mon[24869]: mgrmap e9: y(active, since 16s) 2024-04-14T15:19:40.954 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:40 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:40.832+0000 7fbd44b7c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:19:40.955 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:40 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:40.923+0000 7fbd44b7c000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:19:41.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:41.011+0000 7fbd44b7c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:19:41.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:41.169+0000 7fbd44b7c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:19:41.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:41.252+0000 7fbd44b7c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:19:41.391 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: { 2024-04-14T15:19:41.391 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-14T15:19:41.391 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "available": true, 2024-04-14T15:19:41.392 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-14T15:19:41.392 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-14T15:19:41.392 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: } 2024-04-14T15:19:41.620 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:41 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3148635056' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-14T15:19:41.621 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:41.364+0000 7fbd44b7c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:19:41.621 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:41.454+0000 7fbd44b7c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:19:41.621 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:41.620+0000 7fbd44b7c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:19:42.010 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:41.708+0000 7fbd44b7c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:19:42.013 INFO:teuthology.orchestra.run.smithi016.stderr:Waiting for the mgr to restart... 2024-04-14T15:19:42.013 INFO:teuthology.orchestra.run.smithi016.stderr:Waiting for mgr epoch 9... 2024-04-14T15:19:43.104 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:42.775+0000 7fbd44b7c000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:19:43.104 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:42.909+0000 7fbd44b7c000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:19:43.104 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:43.103+0000 7fbd44b7c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:19:43.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:43.192+0000 7fbd44b7c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:19:43.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:43.333+0000 7fbd44b7c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:19:43.972 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:43.656+0000 7fbd44b7c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:19:43.972 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:43.884+0000 7fbd44b7c000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:19:43.973 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:43.972+0000 7fbd44b7c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:19:44.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:44.147+0000 7fbd44b7c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:19:45.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:45 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:45.087+0000 7fbd44b7c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:19:45.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:45 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:45.394+0000 7fbd44b7c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:19:45.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:45 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:45.491+0000 7fbd44b7c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:19:45.646 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:45 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:19:45.645+0000 7fbd44b7c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:19:46.709 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: { 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: Active manager daemon y restarted 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: Activating manager daemon y 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: osdmap e3: 0 total, 0 up, 0 in 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: mgrmap e10: y(active, starting, since 0.052331s) 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: Manager daemon y is now available 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:46.710 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:19:46.711 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:19:46.711 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-14T15:19:46.711 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: "initialized": true 2024-04-14T15:19:46.711 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: } 2024-04-14T15:19:47.458 INFO:teuthology.orchestra.run.smithi016.stderr:mgr epoch 9 is available 2024-04-14T15:19:47.458 INFO:teuthology.orchestra.run.smithi016.stderr:Generating a dashboard self-signed certificate... 2024-04-14T15:19:47.705 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:47 smithi016 ceph-mon[24869]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-14T15:19:47.705 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:47 smithi016 ceph-mon[24869]: mgrmap e11: y(active, since 1.05504s) 2024-04-14T15:19:47.705 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:47 smithi016 ceph-mon[24869]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-14T15:19:47.705 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:47 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:19:48.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:19:47] ENGINE Bus STARTING 2024-04-14T15:19:48.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:19:47] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:19:48.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:19:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:19:47] ENGINE Bus STARTED 2024-04-14T15:19:48.179 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-14T15:19:48.889 INFO:teuthology.orchestra.run.smithi016.stderr:Creating initial admin user... 2024-04-14T15:19:49.206 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:48 smithi016 ceph-mon[24869]: [14/Apr/2024:15:19:47] ENGINE Bus STARTING 2024-04-14T15:19:49.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:48 smithi016 ceph-mon[24869]: [14/Apr/2024:15:19:47] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:19:49.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:48 smithi016 ceph-mon[24869]: [14/Apr/2024:15:19:47] ENGINE Bus STARTED 2024-04-14T15:19:49.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:48 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:19:49.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:48 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:19:49.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:48 smithi016 ceph-mon[24869]: mgrmap e12: y(active, since 2s) 2024-04-14T15:19:49.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:48 smithi016 ceph-mon[24869]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:19:49.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:48 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:19:49.207 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:48 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:19:49.776 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$eeS/shYFObA.KvHJGhWXaufVA0Ak22Z0yedng80ZvefmckvfP5sPq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713107989, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-14T15:19:50.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:49 smithi016 ceph-mon[24869]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:19:50.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:19:51.170 INFO:teuthology.orchestra.run.smithi016.stderr:Fetching dashboard port number... 2024-04-14T15:19:51.785 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: 8443 2024-04-14T15:19:52.479 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: disabled 2024-04-14T15:19:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:systemctl: inactive 2024-04-14T15:19:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:firewalld.service is not enabled 2024-04-14T15:19:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-14T15:19:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:Ceph Dashboard is now available at: 2024-04-14T15:19:52.489 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-14T15:19:52.489 INFO:teuthology.orchestra.run.smithi016.stderr: URL: https://smithi016.front.sepia.ceph.com:8443/ 2024-04-14T15:19:52.489 INFO:teuthology.orchestra.run.smithi016.stderr: User: admin 2024-04-14T15:19:52.490 INFO:teuthology.orchestra.run.smithi016.stderr: Password: f2ta0ohpjr 2024-04-14T15:19:52.490 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-14T15:19:52.491 INFO:teuthology.orchestra.run.smithi016.stderr:Enabling autotune for osd_memory_target 2024-04-14T15:19:52.718 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:52 smithi016 ceph-mon[24869]: mgrmap e13: y(active, since 6s) 2024-04-14T15:19:52.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:52 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3711679085' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-14T15:19:54.573 INFO:teuthology.orchestra.run.smithi016.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-14T15:19:55.226 INFO:teuthology.orchestra.run.smithi016.stderr:You can access the Ceph CLI with: 2024-04-14T15:19:55.226 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-14T15:19:55.226 INFO:teuthology.orchestra.run.smithi016.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:19:55.226 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-14T15:19:55.226 INFO:teuthology.orchestra.run.smithi016.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-14T15:19:55.226 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-14T15:19:55.226 INFO:teuthology.orchestra.run.smithi016.stderr: ceph telemetry on 2024-04-14T15:19:55.226 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-14T15:19:55.227 INFO:teuthology.orchestra.run.smithi016.stderr:For more information see: 2024-04-14T15:19:55.227 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-14T15:19:55.227 INFO:teuthology.orchestra.run.smithi016.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-14T15:19:55.227 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-14T15:19:55.227 INFO:teuthology.orchestra.run.smithi016.stderr:Bootstrap complete. 2024-04-14T15:19:55.273 INFO:tasks.cephadm:Fetching config... 2024-04-14T15:19:55.273 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-14T15:19:55.273 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-14T15:19:55.290 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-14T15:19:55.290 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-14T15:19:55.290 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-14T15:19:55.345 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-14T15:19:55.345 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-14T15:19:55.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-14T15:19:55.413 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-14T15:19:55.413 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-14T15:19:55.413 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-14T15:19:55.468 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-14T15:19:55.468 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoHt4gx1d74c/02rtMNGRzV0IwgUZdf+nTGLujNob0s1SLXS9laSwZe9UJ1CKzdb3l8bAOlOfig/Qbh3r9rHuU3vQazmL9J7Omifu7UrxX1UsdDvqiRZDrOfGD8K6yfWon0n8cfFtN7cL1YWejFRfOTnhLP1i9qbhDTmUFuH7IbwQubP2ujA+OawgOYKbX5Hd5Le71Rl8S0HzOG6zkfWWM8JuhgE8oTmmZfydfXTvQ2ShGv/2kC2tNDiO18Lh0W1fp8yk6T0Xrbzqx4owsDZ4JN0vk/pqwTExvYt55nsSWx4U6AVs4R+303sXorc6jJZdXax7mbcUhd48GlAvB3rC7lut9H5e1IJQlMF8M2xHKxnDDD3v0uzTmPiA2zjfKOQGFeDg5F259k+c2lnHoBF8LjAh7HBayjGeCa8JUysqzbwVnkx5YAPXDLOmj8A/qmJsI0GFxirBbxURH6LbS6SqGZLbxBEh3BFqjfgIqiXHz+zDBbGiukoce8/8v3iEt9T0= ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-14T15:19:55.549 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoHt4gx1d74c/02rtMNGRzV0IwgUZdf+nTGLujNob0s1SLXS9laSwZe9UJ1CKzdb3l8bAOlOfig/Qbh3r9rHuU3vQazmL9J7Omifu7UrxX1UsdDvqiRZDrOfGD8K6yfWon0n8cfFtN7cL1YWejFRfOTnhLP1i9qbhDTmUFuH7IbwQubP2ujA+OawgOYKbX5Hd5Le71Rl8S0HzOG6zkfWWM8JuhgE8oTmmZfydfXTvQ2ShGv/2kC2tNDiO18Lh0W1fp8yk6T0Xrbzqx4owsDZ4JN0vk/pqwTExvYt55nsSWx4U6AVs4R+303sXorc6jJZdXax7mbcUhd48GlAvB3rC7lut9H5e1IJQlMF8M2xHKxnDDD3v0uzTmPiA2zjfKOQGFeDg5F259k+c2lnHoBF8LjAh7HBayjGeCa8JUysqzbwVnkx5YAPXDLOmj8A/qmJsI0GFxirBbxURH6LbS6SqGZLbxBEh3BFqjfgIqiXHz+zDBbGiukoce8/8v3iEt9T0= ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:19:55.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoHt4gx1d74c/02rtMNGRzV0IwgUZdf+nTGLujNob0s1SLXS9laSwZe9UJ1CKzdb3l8bAOlOfig/Qbh3r9rHuU3vQazmL9J7Omifu7UrxX1UsdDvqiRZDrOfGD8K6yfWon0n8cfFtN7cL1YWejFRfOTnhLP1i9qbhDTmUFuH7IbwQubP2ujA+OawgOYKbX5Hd5Le71Rl8S0HzOG6zkfWWM8JuhgE8oTmmZfydfXTvQ2ShGv/2kC2tNDiO18Lh0W1fp8yk6T0Xrbzqx4owsDZ4JN0vk/pqwTExvYt55nsSWx4U6AVs4R+303sXorc6jJZdXax7mbcUhd48GlAvB3rC7lut9H5e1IJQlMF8M2xHKxnDDD3v0uzTmPiA2zjfKOQGFeDg5F259k+c2lnHoBF8LjAh7HBayjGeCa8JUysqzbwVnkx5YAPXDLOmj8A/qmJsI0GFxirBbxURH6LbS6SqGZLbxBEh3BFqjfgIqiXHz+zDBbGiukoce8/8v3iEt9T0= ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-14T15:19:55.609 INFO:teuthology.orchestra.run.smithi196.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoHt4gx1d74c/02rtMNGRzV0IwgUZdf+nTGLujNob0s1SLXS9laSwZe9UJ1CKzdb3l8bAOlOfig/Qbh3r9rHuU3vQazmL9J7Omifu7UrxX1UsdDvqiRZDrOfGD8K6yfWon0n8cfFtN7cL1YWejFRfOTnhLP1i9qbhDTmUFuH7IbwQubP2ujA+OawgOYKbX5Hd5Le71Rl8S0HzOG6zkfWWM8JuhgE8oTmmZfydfXTvQ2ShGv/2kC2tNDiO18Lh0W1fp8yk6T0Xrbzqx4owsDZ4JN0vk/pqwTExvYt55nsSWx4U6AVs4R+303sXorc6jJZdXax7mbcUhd48GlAvB3rC7lut9H5e1IJQlMF8M2xHKxnDDD3v0uzTmPiA2zjfKOQGFeDg5F259k+c2lnHoBF8LjAh7HBayjGeCa8JUysqzbwVnkx5YAPXDLOmj8A/qmJsI0GFxirBbxURH6LbS6SqGZLbxBEh3BFqjfgIqiXHz+zDBbGiukoce8/8v3iEt9T0= ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:19:55.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-14T15:19:55.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:55 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2076573185' entity='client.admin' 2024-04-14T15:19:58.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:58 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:19:58.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:58 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:58 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:58 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:58.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:58 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:19:59.008 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-14T15:19:59.009 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-14T15:19:59.560 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:59 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/453623859' entity='client.admin' 2024-04-14T15:19:59.560 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:19:59.560 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:19:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:01.981 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi196 2024-04-14T15:20:01.981 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-14T15:20:01.981 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.conf 2024-04-14T15:20:01.999 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-14T15:20:01.999 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:20:02.057 INFO:tasks.cephadm:Adding host smithi196 to orchestrator... 2024-04-14T15:20:02.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch host add smithi196 2024-04-14T15:20:02.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:02 smithi016 ceph-mon[24869]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:02.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:02 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:02.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:02 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:02 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:02 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:02 smithi016 ceph-mon[24869]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:20:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:02 smithi016 ceph-mon[24869]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:20:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:02 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:02 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:05.048 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi196' with addr '172.21.15.196' 2024-04-14T15:20:05.048 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:04 smithi016 ceph-mon[24869]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:05.048 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:04 smithi016 ceph-mon[24869]: Deploying cephadm binary to smithi196 2024-04-14T15:20:05.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch host ls --format=json 2024-04-14T15:20:06.223 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:06 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:06.223 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:06 smithi016 ceph-mon[24869]: Added host smithi196 2024-04-14T15:20:06.223 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:06 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:06.223 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:06 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:06.223 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:06 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:06.223 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:06 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:06.223 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:06 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:07.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:07 smithi016 ceph-mon[24869]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:07.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:07 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:07.506 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:20:07.506 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.196", "hostname": "smithi196", "labels": [], "status": ""}] 2024-04-14T15:20:08.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:08 smithi016 ceph-mon[24869]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:20:08.152 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-14T15:20:08.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd crush tunables default 2024-04-14T15:20:09.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:09 smithi016 ceph-mon[24869]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:09.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:09.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:09.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:09.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:10.643 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-04-14T15:20:10.990 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:10 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1666535812' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-14T15:20:11.252 INFO:tasks.cephadm:Adding mon.a on smithi016 2024-04-14T15:20:11.252 INFO:tasks.cephadm:Adding mon.c on smithi016 2024-04-14T15:20:11.252 INFO:tasks.cephadm:Adding mon.b on smithi196 2024-04-14T15:20:11.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch apply mon '3;smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b' 2024-04-14T15:20:12.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:11 smithi016 ceph-mon[24869]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:12.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:11 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1666535812' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-14T15:20:12.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:11 smithi016 ceph-mon[24869]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:20:12.982 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mon update... 2024-04-14T15:20:13.595 DEBUG:teuthology.orchestra.run.smithi016:mon.c> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.c.service 2024-04-14T15:20:13.597 DEBUG:teuthology.orchestra.run.smithi196:mon.b> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.b.service 2024-04-14T15:20:13.599 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-14T15:20:13.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T15:20:13.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:13 smithi016 ceph-mon[24869]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:13.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:13 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:13 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:13 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:13 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:13 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:13 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:13 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:20:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:13 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:14.973 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:14 smithi016 ceph-mon[24869]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:14.973 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:14 smithi016 ceph-mon[24869]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b;count:3 2024-04-14T15:20:14.974 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:14 smithi016 ceph-mon[24869]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:20:14.974 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:14 smithi016 ceph-mon[24869]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:20:14.974 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:14 smithi016 ceph-mon[24869]: Deploying daemon mon.c on smithi016 2024-04-14T15:20:15.388 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-14T15:20:15.388 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"3ca5917a-fa72-11ee-bc8e-c7b262605968","modified":"2024-04-14T15:18:56.351711Z","created":"2024-04-14T15:18:56.351711Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T15:20:15.390 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-04-14T15:20:15.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:15 smithi016 ceph-mon[24869]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:15.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:15 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/2316961114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.527 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 podman[29498]: 2024-04-14 15:20:16.370200591 +0000 UTC m=+0.018103257 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-14T15:20:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 podman[29498]: 2024-04-14 15:20:16.616662279 +0000 UTC m=+0.264564944 container init 8629057bb8e95e9c24d1fd8fbcef970c0392c9580d56cccdf07d9a0ff1edf9d7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, RELEASE=HEAD, release=754, ceph=True, GIT_CLEAN=True, vcs-type=git, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0) 2024-04-14T15:20:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 podman[29498]: 2024-04-14 15:20:16.619563948 +0000 UTC m=+0.267466623 container start 8629057bb8e95e9c24d1fd8fbcef970c0392c9580d56cccdf07d9a0ff1edf9d7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.buildah.version=1.19.8, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, ceph=True, name=centos-stream, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, architecture=x86_64, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., vcs-type=git) 2024-04-14T15:20:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T15:20:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-14T15:20:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: pidfile_write: ignore empty --pid-file 2024-04-14T15:20:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: load: jerasure load: lrc 2024-04-14T15:20:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: RocksDB version: 6.15.5 2024-04-14T15:20:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-14T15:20:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Compile date Apr 18 2022 2024-04-14T15:20:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: DB SUMMARY 2024-04-14T15:20:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: DB Session ID: AC6XLXR29P8GMI5T9PEI 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: CURRENT file: CURRENT 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.error_if_exists: 0 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.create_if_missing: 0 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.paranoid_checks: 1 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.env: 0x559f6c2c6860 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.fs: Posix File System 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.info_log: 0x559f6d06ddc0 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.statistics: (nil) 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.use_fsync: 0 2024-04-14T15:20:16.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_log_file_size: 0 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.allow_fallocate: 1 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.use_direct_reads: 0 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.db_log_dir: 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T15:20:16.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T15:20:16.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T15:20:16.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T15:20:16.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T15:20:16.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.write_buffer_manager: 0x559f6d15e2d0 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.unordered_write: 0 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T15:20:16.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.row_cache: None 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.wal_filter: None 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.preserve_deletes: 0 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.two_write_queues: 0 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.atomic_flush: 0 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.log_readahead_size: 0 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T15:20:16.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_background_jobs: 2 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_background_compactions: -1 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_subcompactions: 1 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T15:20:16.868 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_open_files: -1 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_background_flushes: -1 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Compression algorithms supported: 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: kZSTD supported: 0 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: kXpressCompression supported: 0 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: kLZ4Compression supported: 1 2024-04-14T15:20:16.869 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: kBZip2Compression supported: 0 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: kZlibCompression supported: 1 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: kSnappyCompression supported: 1 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.merge_operator: 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_filter: None 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559f6d039d00) 2024-04-14T15:20:16.870 INFO:journalctl@ceph.mon.c.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_type: 0 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: data_block_index_type: 0 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_shortening: 1 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: hash_index_allow_collision: 1 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: checksum: 1 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: no_block_cache: 0 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache: 0x559f6d0a4170 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_options: 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: capacity : 536870912 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: num_shard_bits : 4 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: strict_capacity_limit : 0 2024-04-14T15:20:16.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_compressed: (nil) 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: persistent_cache: (nil) 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_size: 4096 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_size_deviation: 10 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_restart_interval: 16 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_block_restart_interval: 1 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: metadata_block_size: 4096 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: partition_filters: 0 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: use_delta_encoding: 1 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: whole_key_filtering: 1 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: verify_compression: 0 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: format_version: 4 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: enable_index_compression: 1 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_align: 0 2024-04-14T15:20:16.872 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compression: NoCompression 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.num_levels: 7 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T15:20:16.873 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T15:20:16.874 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.arena_block_size: 4194304 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T15:20:16.875 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.table_properties_collectors: 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.inplace_update_support: 0 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.bloom_locality: 0 2024-04-14T15:20:16.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.max_successive_merges: 0 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.ttl: 2592000 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.enable_blob_files: false 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.min_blob_size: 0 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108016646165, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-14T15:20:16.877 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108016646460, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713108016, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "851c5149-c7e0-4c0b-871d-9137b08fa5ae", "db_session_id": "AC6XLXR29P8GMI5T9PEI"}} 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108016646787, "job": 1, "event": "recovery_finished"} 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x559f6d08aa80 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: DB pointer 0x559f6d09a000 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** DB Stats ** 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T15:20:16.878 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:20:16.879 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:20:16.879 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T15:20:16.879 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-14T15:20:16.879 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:20:16.879 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:20:16.879 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:20:16.879 INFO:journalctl@ceph.mon.c.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:20:16.879 INFO:journalctl@ceph.mon.c.smithi016.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:20:16.879 INFO:journalctl@ceph.mon.c.smithi016.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:20:16.880 INFO:journalctl@ceph.mon.c.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T15:20:16.881 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: using public_addrv [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: starting mon.c rank -1 at public addrs [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] at bind addrs [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(???) e0 preinit fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).mds e1 new map 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout: e1 2024-04-14T15:20:16.882 INFO:journalctl@ceph.mon.c.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout: legacy client fscid: -1 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout: No filesystems configured 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:20:16.883 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mkfs 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: monmap e1: 1 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: fsmap 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: osdmap e1: 0 total, 0 up, 0 in 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e1: no daemons active 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1362727343' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/47750983' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/47750983' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/777671436' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: monmap e1: 1 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: fsmap 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: osdmap e1: 0 total, 0 up, 0 in 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e1: no daemons active 2024-04-14T15:20:16.884 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/16094027' entity='client.admin' 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3225196975' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Activating manager daemon y 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e2: y(active, starting, since 0.0028212s) 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Manager daemon y is now available 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14100 172.21.15.16:0/711836865' entity='mgr.y' 2024-04-14T15:20:16.885 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e3: y(active, since 1.00716s) 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e4: y(active, since 2s) 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4259571193' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4275195754' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4275195754' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1823767811' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1823767811' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e5: y(active, since 5s) 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1653464846' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Active manager daemon y restarted 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Activating manager daemon y 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: osdmap e2: 0 total, 0 up, 0 in 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e6: y(active, starting, since 0.0522864s) 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:20:16.886 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Manager daemon y is now available 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e7: y(active, since 1.05476s) 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e8: y(active, since 2s) 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.887 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: [14/Apr/2024:15:19:25] ENGINE Bus STARTING 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: [14/Apr/2024:15:19:25] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: [14/Apr/2024:15:19:25] ENGINE Bus STARTED 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Generating ssh key... 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Deploying cephadm binary to smithi016 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Added host smithi016 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.888 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:16.889 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Saving service mon spec with placement count:5 2024-04-14T15:20:16.889 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.889 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:16.889 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Saving service mgr spec with placement count:2 2024-04-14T15:20:16.889 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.889 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.889 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.889 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2550926075' entity='client.admin' 2024-04-14T15:20:16.889 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.889 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3102626295' entity='client.admin' 2024-04-14T15:20:16.889 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14116 172.21.15.16:0/3169308853' entity='mgr.y' 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4151726119' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4151726119' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e9: y(active, since 16s) 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3148635056' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Active manager daemon y restarted 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Activating manager daemon y 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: osdmap e3: 0 total, 0 up, 0 in 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e10: y(active, starting, since 0.052331s) 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:20:16.890 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Manager daemon y is now available 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e11: y(active, since 1.05504s) 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: [14/Apr/2024:15:19:47] ENGINE Bus STARTING 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: [14/Apr/2024:15:19:47] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: [14/Apr/2024:15:19:47] ENGINE Bus STARTED 2024-04-14T15:20:16.891 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e12: y(active, since 2s) 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mgrmap e13: y(active, since 6s) 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3711679085' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2076573185' entity='client.admin' 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/453623859' entity='client.admin' 2024-04-14T15:20:16.892 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Deploying cephadm binary to smithi196 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Added host smithi196 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.893 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1666535812' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-14T15:20:16.894 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1666535812' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Saving service mon spec with placement smithi016:172.21.15.16=a;smithi016:[v2:172.21.15.16:3301,v1:172.21.15.16:6790]=c;smithi196:172.21.15.196=b;count:3 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: Deploying daemon mon.c on smithi016 2024-04-14T15:20:16.895 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:16.896 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/2316961114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:20:16.896 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-14T15:20:16.896 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 ceph-mon[29517]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-14T15:20:16.896 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 bash[29498]: 8629057bb8e95e9c24d1fd8fbcef970c0392c9580d56cccdf07d9a0ff1edf9d7 2024-04-14T15:20:16.896 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:16 smithi016 systemd[1]: Started Ceph mon.c for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:20:16.906 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-14T15:20:16.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T15:20:22.012 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-14T15:20:22.012 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":2,"fsid":"3ca5917a-fa72-11ee-bc8e-c7b262605968","modified":"2024-04-14T15:20:16.665553Z","created":"2024-04-14T15:18:56.351711Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3301","nonce":0},{"type":"v1","addr":"172.21.15.16:6790","nonce":0}]},"addr":"172.21.15.16:6790/0","public_addr":"172.21.15.16:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-14T15:20:22.014 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 2 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: mon.a calling monitor election 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: mon.c calling monitor election 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: monmap e2: 2 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0]} 2024-04-14T15:20:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: fsmap 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: mgrmap e13: y(active, since 36s) 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: overall HEALTH_OK 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: mon.a calling monitor election 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: mon.c calling monitor election 2024-04-14T15:20:22.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: monmap e2: 2 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],c=[v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0]} 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: fsmap 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: mgrmap e13: y(active, since 36s) 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: overall HEALTH_OK 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:20:22.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:21 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:23.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:22 smithi016 ceph-mon[24869]: Deploying daemon mon.b on smithi196 2024-04-14T15:20:23.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:22 smithi016 ceph-mon[24869]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:23.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:22 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/1345850086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:20:23.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:22 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:23.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:22 smithi016 ceph-mon[29517]: Deploying daemon mon.b on smithi196 2024-04-14T15:20:23.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:22 smithi016 ceph-mon[29517]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:23.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:22 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/1345850086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:20:23.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:22 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:24.204 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-14T15:20:24.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T15:20:25.631 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:25 smithi196 systemd[1]: Starting Ceph mon.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:20:26.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:25 smithi016 ceph-mon[24869]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:26.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:25 smithi016 ceph-mon[29517]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:26.591 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-14T15:20:26.591 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":2,"fsid":"3ca5917a-fa72-11ee-bc8e-c7b262605968","modified":"2024-04-14T15:20:16.665553Z","created":"2024-04-14T15:18:56.351711Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3301","nonce":0},{"type":"v1","addr":"172.21.15.16:6790","nonce":0}]},"addr":"172.21.15.16:6790/0","public_addr":"172.21.15.16:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-14T15:20:26.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 podman[25664]: 2024-04-14 15:20:26.503081654 +0000 UTC m=+0.101433562 container create f933c5fa37fdc610b50da82a90846d439156675daec5bd0ded89ed51a550192b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b, RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, release=754, CEPH_POINT_RELEASE=-17.2.0, ceph=True, name=centos-stream, io.openshift.expose-services=, architecture=x86_64, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-14T15:20:26.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 podman[25664]: 2024-04-14 15:20:26.419894975 +0000 UTC m=+0.018246875 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-14T15:20:26.593 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 2 2024-04-14T15:20:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 podman[25664]: 2024-04-14 15:20:26.688892607 +0000 UTC m=+0.287244508 container init f933c5fa37fdc610b50da82a90846d439156675daec5bd0ded89ed51a550192b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vcs-type=git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, version=8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-14T15:20:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 podman[25664]: 2024-04-14 15:20:26.69205383 +0000 UTC m=+0.290405731 container start f933c5fa37fdc610b50da82a90846d439156675daec5bd0ded89ed51a550192b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, distribution-scope=public, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, release=754, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8) 2024-04-14T15:20:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T15:20:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-14T15:20:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: pidfile_write: ignore empty --pid-file 2024-04-14T15:20:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: load: jerasure load: lrc 2024-04-14T15:20:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: RocksDB version: 6.15.5 2024-04-14T15:20:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-14T15:20:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Compile date Apr 18 2022 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: DB SUMMARY 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: DB Session ID: BDSHCEUJ9S9NLV6NKBFX 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: CURRENT file: CURRENT 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.error_if_exists: 0 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.create_if_missing: 0 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.paranoid_checks: 1 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.env: 0x563de94c6860 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.fs: Posix File System 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.info_log: 0x563dea71bdc0 2024-04-14T15:20:26.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.statistics: (nil) 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.use_fsync: 0 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_log_file_size: 0 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.allow_fallocate: 1 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.use_direct_reads: 0 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.db_log_dir: 2024-04-14T15:20:26.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-04-14T15:20:26.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T15:20:26.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T15:20:26.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T15:20:26.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T15:20:26.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T15:20:26.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T15:20:26.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T15:20:26.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T15:20:26.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.write_buffer_manager: 0x563dea80c240 2024-04-14T15:20:26.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T15:20:26.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-14T15:20:26.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T15:20:26.919 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.unordered_write: 0 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.row_cache: None 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.wal_filter: None 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.preserve_deletes: 0 2024-04-14T15:20:26.920 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.two_write_queues: 0 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.atomic_flush: 0 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.log_readahead_size: 0 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_background_jobs: 2 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_background_compactions: -1 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_subcompactions: 1 2024-04-14T15:20:26.921 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_open_files: -1 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_background_flushes: -1 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Compression algorithms supported: 2024-04-14T15:20:26.922 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: kZSTD supported: 0 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: kXpressCompression supported: 0 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: kLZ4Compression supported: 1 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: kBZip2Compression supported: 0 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: kZlibCompression supported: 1 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: kSnappyCompression supported: 1 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.merge_operator: 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_filter: None 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T15:20:26.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563dea6e7d00) 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks: 1 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_top_level_index_and_filter: 1 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_type: 0 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_index_type: 0 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_shortening: 1 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: hash_index_allow_collision: 1 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: checksum: 1 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: no_block_cache: 0 2024-04-14T15:20:26.924 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache: 0x563dea752170 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_name: BinnedLRUCache 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_options: 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: capacity : 536870912 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_shard_bits : 4 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: strict_capacity_limit : 0 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: high_pri_pool_ratio: 0.000 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_compressed: (nil) 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: persistent_cache: (nil) 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size: 4096 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size_deviation: 10 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_restart_interval: 16 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_block_restart_interval: 1 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: metadata_block_size: 4096 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: partition_filters: 0 2024-04-14T15:20:26.925 INFO:journalctl@ceph.mon.b.smithi196.stdout: use_delta_encoding: 1 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout: whole_key_filtering: 1 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout: verify_compression: 0 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout: read_amp_bytes_per_bit: 0 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout: format_version: 4 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_index_compression: 1 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_align: 0 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compression: NoCompression 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.num_levels: 7 2024-04-14T15:20:26.926 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T15:20:26.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T15:20:26.928 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.arena_block_size: 4194304 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T15:20:26.929 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.table_properties_collectors: 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.inplace_update_support: 0 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.bloom_locality: 0 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.max_successive_merges: 0 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.ttl: 2592000 2024-04-14T15:20:26.930 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.enable_blob_files: false 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.min_blob_size: 0 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108026718051, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108026718350, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713108026, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "67978c58-7d42-4141-8e54-56126af8cb02", "db_session_id": "BDSHCEUJ9S9NLV6NKBFX"}} 2024-04-14T15:20:26.931 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108026718654, "job": 1, "event": "recovery_finished"} 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x563dea738a80 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: DB pointer 0x563dea748000 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** DB Stats ** 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-04-14T15:20:26.932 INFO:journalctl@ceph.mon.b.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:20:26.933 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:20:26.934 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:20:26.934 INFO:journalctl@ceph.mon.b.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:20:26.934 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:20:26.934 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:20:26.934 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:20:26.934 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-04-14T15:20:26.934 INFO:journalctl@ceph.mon.b.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:20:26.934 INFO:journalctl@ceph.mon.b.smithi196.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:20:26.934 INFO:journalctl@ceph.mon.b.smithi196.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:26.934 INFO:journalctl@ceph.mon.b.smithi196.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:26.934 INFO:journalctl@ceph.mon.b.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:20:26.935 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: using public_addr v2:172.21.15.196:0/0 -> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: starting mon.b rank -1 at public addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] at bind addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(???) e0 preinit fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).mds e1 new map 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout: e1 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout: legacy client fscid: -1 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout: No filesystems configured 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-14T15:20:26.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: Deploying daemon mon.b on smithi196 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: from='client.? 172.21.15.196:0/1345850086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: from='client.? 172.21.15.196:0/2255292315' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 ceph-mon[25696]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 bash[25664]: f933c5fa37fdc610b50da82a90846d439156675daec5bd0ded89ed51a550192b 2024-04-14T15:20:26.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:26 smithi196 systemd[1]: Started Ceph mon.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:20:27.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:26 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/2255292315' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:20:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:26 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/2255292315' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T15:20:28.297 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-14T15:20:28.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T15:20:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:20:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: mon.a calling monitor election 2024-04-14T15:20:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: mon.c calling monitor election 2024-04-14T15:20:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0]} 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: fsmap 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: mgrmap e13: y(active, since 46s) 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: mon.b (rank 2) addr [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] is down (out of quorum) 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:32.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: mon.a calling monitor election 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: mon.c calling monitor election 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:32.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0]} 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: fsmap 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: mgrmap e13: y(active, since 46s) 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: mon.b (rank 2) addr [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] is down (out of quorum) 2024-04-14T15:20:32.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:32.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:32.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:32.115 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:33.031 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:32 smithi016 ceph-mon[24869]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:33.031 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:33.031 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:33.031 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:33.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:32 smithi016 ceph-mon[29517]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:33.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:33.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:33.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:33.111 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-04-14T15:20:33.111 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":3,"fsid":"3ca5917a-fa72-11ee-bc8e-c7b262605968","modified":"2024-04-14T15:20:26.737782Z","created":"2024-04-14T15:18:56.351711Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3301","nonce":0},{"type":"v1","addr":"172.21.15.16:6790","nonce":0}]},"addr":"172.21.15.16:6790/0","public_addr":"172.21.15.16:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-14T15:20:33.113 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 3 2024-04-14T15:20:33.587 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-14T15:20:33.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph config generate-minimal-conf 2024-04-14T15:20:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:20:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: mon.a calling monitor election 2024-04-14T15:20:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: mon.c calling monitor election 2024-04-14T15:20:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:20:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0]} 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: fsmap 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: mgrmap e13: y(active, since 46s) 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: mon.b (rank 2) addr [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] is down (out of quorum) 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:34.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:34.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:34.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:34.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:34.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: mon.b calling monitor election 2024-04-14T15:20:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: mon.b calling monitor election 2024-04-14T15:20:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: mon.c calling monitor election 2024-04-14T15:20:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: mon.a calling monitor election 2024-04-14T15:20:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:20:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0]} 2024-04-14T15:20:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: fsmap 2024-04-14T15:20:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:20:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: mgrmap e13: y(active, since 48s) 2024-04-14T15:20:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-14T15:20:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: Cluster is now healthy 2024-04-14T15:20:34.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: overall HEALTH_OK 2024-04-14T15:20:34.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:34.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:20:34.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:20:34.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:34 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:35.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: mon.b calling monitor election 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: mon.b calling monitor election 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: mon.c calling monitor election 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: mon.a calling monitor election 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0]} 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: fsmap 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: mgrmap e13: y(active, since 48s) 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: Cluster is now healthy 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: overall HEALTH_OK 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:35.102 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: mon.b calling monitor election 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: mon.b calling monitor election 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: mon.c calling monitor election 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: mon.a calling monitor election 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: monmap e3: 3 mons at {a=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0]} 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: fsmap 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: mgrmap e13: y(active, since 48s) 2024-04-14T15:20:35.103 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-14T15:20:35.104 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: Cluster is now healthy 2024-04-14T15:20:35.104 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: overall HEALTH_OK 2024-04-14T15:20:35.104 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:35.104 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:20:35.104 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:20:35.104 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:34 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:35.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:35 smithi016 ceph-mon[24869]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:35.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:35 smithi016 ceph-mon[24869]: Reconfiguring mon.c (monmap changed)... 2024-04-14T15:20:35.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:35 smithi016 ceph-mon[24869]: Reconfiguring daemon mon.c on smithi016 2024-04-14T15:20:35.685 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:35 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:35.685 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:35 smithi016 ceph-mon[29517]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:35.685 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:35 smithi016 ceph-mon[29517]: Reconfiguring mon.c (monmap changed)... 2024-04-14T15:20:35.685 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:35 smithi016 ceph-mon[29517]: Reconfiguring daemon mon.c on smithi016 2024-04-14T15:20:35.685 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:35 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:35.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:35 smithi196 ceph-mon[25696]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:35.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:35 smithi196 ceph-mon[25696]: Reconfiguring mon.c (monmap changed)... 2024-04-14T15:20:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:35 smithi196 ceph-mon[25696]: Reconfiguring daemon mon.c on smithi016 2024-04-14T15:20:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:35 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:20:36.603 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:20:36.603 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-04-14T15:20:36.603 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:20:36.603 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] 2024-04-14T15:20:36.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:36 smithi016 ceph-mon[24869]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:36.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:36 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:36.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:36 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1419732449' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:36.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:36 smithi016 ceph-mon[29517]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:36.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:36 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:36.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:36 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1419732449' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:37.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:36 smithi196 ceph-mon[25696]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:37.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:36 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:37.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:36 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1419732449' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:37.477 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-14T15:20:37.477 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-14T15:20:37.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-04-14T15:20:37.508 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-14T15:20:37.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:20:37.575 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-14T15:20:37.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.conf 2024-04-14T15:20:37.604 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-14T15:20:37.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:20:37.670 INFO:tasks.cephadm:Adding mgr.y on smithi016 2024-04-14T15:20:37.671 INFO:tasks.cephadm:Adding mgr.x on smithi196 2024-04-14T15:20:37.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch apply mgr '2;smithi016=y;smithi196=x' 2024-04-14T15:20:38.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:37 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:38.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:37 smithi196 ceph-mon[25696]: Reconfiguring mon.b (monmap changed)... 2024-04-14T15:20:38.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:37 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:20:38.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:37 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:20:38.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:37 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:38.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:37 smithi196 ceph-mon[25696]: Reconfiguring daemon mon.b on smithi196 2024-04-14T15:20:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[24869]: Reconfiguring mon.b (monmap changed)... 2024-04-14T15:20:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:20:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:20:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[24869]: Reconfiguring daemon mon.b on smithi196 2024-04-14T15:20:38.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:38.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[29517]: Reconfiguring mon.b (monmap changed)... 2024-04-14T15:20:38.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:20:38.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:20:38.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:38.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:37 smithi016 ceph-mon[29517]: Reconfiguring daemon mon.b on smithi196 2024-04-14T15:20:39.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:39 smithi196 ceph-mon[25696]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:39.321 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:39 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:39.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:39 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:39.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:39 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:39.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:39 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:39.322 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:39 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:39.609 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mgr update... 2024-04-14T15:20:39.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[29517]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:39.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:39.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:39.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:39.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:39.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:39.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[24869]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:39.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:39.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:39.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:39.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:39.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:39 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:40.239 DEBUG:teuthology.orchestra.run.smithi196:mgr.x> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.x.service 2024-04-14T15:20:40.241 INFO:tasks.cephadm:Deploying OSDs... 2024-04-14T15:20:40.241 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-14T15:20:40.241 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-04-14T15:20:40.260 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-14T15:20:40.260 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-04-14T15:20:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-14T15:20:40.317 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:20:40.317 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-14T15:20:40.317 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:20:40.318 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:20:40.318 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-14 15:20:16.191366963 +0000 2024-04-14T15:20:40.318 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-14 15:18:00.099342711 +0000 2024-04-14T15:20:40.318 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-14 15:18:00.099342711 +0000 2024-04-14T15:20:40.318 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-14 15:18:00.099342711 +0000 2024-04-14T15:20:40.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-14T15:20:40.387 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-14T15:20:40.387 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-14T15:20:40.387 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000113169 s, 4.5 MB/s 2024-04-14T15:20:40.388 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-14T15:20:40.445 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-04-14T15:20:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-14T15:20:40.502 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:20:40.502 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-14T15:20:40.503 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:20:40.503 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:20:40.503 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-14 15:20:16.191366963 +0000 2024-04-14T15:20:40.503 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-14 15:18:00.447337554 +0000 2024-04-14T15:20:40.503 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-14 15:18:00.447337554 +0000 2024-04-14T15:20:40.503 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-14 15:18:00.447337554 +0000 2024-04-14T15:20:40.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-14T15:20:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-14T15:20:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-14T15:20:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.00011302 s, 4.5 MB/s 2024-04-14T15:20:40.571 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-14T15:20:40.627 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-04-14T15:20:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-14T15:20:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:20:40.684 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-14T15:20:40.684 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:20:40.684 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:20:40.684 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-14 15:20:16.192366949 +0000 2024-04-14T15:20:40.684 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-14 15:18:00.779332635 +0000 2024-04-14T15:20:40.684 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-14 15:18:00.779332635 +0000 2024-04-14T15:20:40.684 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-14 15:18:00.779332635 +0000 2024-04-14T15:20:40.685 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-14T15:20:40.748 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi016=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: Saving service mgr spec with placement smithi016=y;smithi196=x;count:2 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[24869]: Deploying daemon mgr.x on smithi196 2024-04-14T15:20:40.749 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi016=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:40.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: Saving service mgr spec with placement smithi016=y;smithi196=x;count:2 2024-04-14T15:20:40.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:40.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:40.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:40.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:40.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:40.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:20:40.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-14T15:20:40.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:20:40.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:40.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:40 smithi016 ceph-mon[29517]: Deploying daemon mgr.x on smithi196 2024-04-14T15:20:40.751 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-14T15:20:40.751 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-14T15:20:40.751 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 9.2081e-05 s, 5.6 MB/s 2024-04-14T15:20:40.752 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-14T15:20:40.808 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-04-14T15:20:40.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi016=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:40.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: Saving service mgr spec with placement smithi016=y;smithi196=x;count:2 2024-04-14T15:20:40.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:40.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:40.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:40.836 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:40.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:40.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:20:40.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-14T15:20:40.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:20:40.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:40.837 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:40 smithi196 ceph-mon[25696]: Deploying daemon mgr.x on smithi196 2024-04-14T15:20:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-14T15:20:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:20:40.863 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-14T15:20:40.863 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:20:40.863 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:20:40.863 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-14 15:20:16.192366949 +0000 2024-04-14T15:20:40.863 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-14 15:18:01.110327731 +0000 2024-04-14T15:20:40.863 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-14 15:18:01.110327731 +0000 2024-04-14T15:20:40.863 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-14 15:18:01.110327731 +0000 2024-04-14T15:20:40.864 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-14T15:20:40.929 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-14T15:20:40.929 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-14T15:20:40.929 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000105064 s, 4.9 MB/s 2024-04-14T15:20:40.930 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-14T15:20:40.985 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-14T15:20:40.986 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-04-14T15:20:41.001 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-14T15:20:41.002 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-04-14T15:20:41.056 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-14T15:20:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:20:41.057 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-14T15:20:41.057 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:20:41.057 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:20:41.057 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-04-14 15:20:25.438765271 +0000 2024-04-14T15:20:41.057 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-04-14 15:17:59.321945352 +0000 2024-04-14T15:20:41.057 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-04-14 15:17:59.321945352 +0000 2024-04-14T15:20:41.057 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-04-14 15:17:59.321945352 +0000 2024-04-14T15:20:41.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-14T15:20:41.123 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-04-14T15:20:41.123 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-04-14T15:20:41.123 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000192612 s, 2.7 MB/s 2024-04-14T15:20:41.124 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-14T15:20:41.179 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-04-14T15:20:41.234 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-14T15:20:41.234 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:20:41.234 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-14T15:20:41.234 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:20:41.234 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:20:41.234 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-04-14 15:20:25.439765250 +0000 2024-04-14T15:20:41.234 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-04-14 15:17:59.675937866 +0000 2024-04-14T15:20:41.234 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-04-14 15:17:59.675937866 +0000 2024-04-14T15:20:41.234 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-04-14 15:17:59.675937866 +0000 2024-04-14T15:20:41.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-14T15:20:41.299 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-04-14T15:20:41.299 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-04-14T15:20:41.299 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000185615 s, 2.8 MB/s 2024-04-14T15:20:41.300 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-14T15:20:41.356 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-04-14T15:20:41.411 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-14T15:20:41.411 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:20:41.411 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-14T15:20:41.412 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:20:41.412 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:20:41.412 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-04-14 15:20:25.439765250 +0000 2024-04-14T15:20:41.412 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-04-14 15:18:00.034930274 +0000 2024-04-14T15:20:41.412 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-04-14 15:18:00.034930274 +0000 2024-04-14T15:20:41.412 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-04-14 15:18:00.034930274 +0000 2024-04-14T15:20:41.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-14T15:20:41.544 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-04-14T15:20:41.544 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-04-14T15:20:41.544 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000207573 s, 2.5 MB/s 2024-04-14T15:20:41.545 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-14T15:20:41.560 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-04-14T15:20:41.617 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-14T15:20:41.617 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T15:20:41.618 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-14T15:20:41.618 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T15:20:41.618 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T15:20:41.618 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-04-14 15:20:41.504436074 +0000 2024-04-14T15:20:41.618 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-04-14 15:18:00.398922577 +0000 2024-04-14T15:20:41.618 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-04-14 15:18:00.398922577 +0000 2024-04-14T15:20:41.618 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-04-14 15:18:00.398922577 +0000 2024-04-14T15:20:41.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-14T15:20:41.712 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-04-14T15:20:41.712 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-04-14T15:20:41.712 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000226685 s, 2.3 MB/s 2024-04-14T15:20:41.713 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-14T15:20:41.728 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-04-14T15:20:41.728 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-14T15:20:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:41 smithi196 ceph-mon[25696]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:41.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:41 smithi196 systemd[1]: Starting Ceph mgr.x for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:20:42.037 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:41 smithi016 ceph-mon[24869]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:42.037 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:41 smithi016 ceph-mon[29517]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:42.243 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:41 smithi196 podman[26869]: 2024-04-14 15:20:41.941473241 +0000 UTC m=+0.109736811 container create 728647a3cffb45d60dafb7e81a8f3fb28ea1b74b6c087e5a28f2668c28778738 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., ceph=True, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, maintainer=Guillaume Abrioux , name=centos-stream, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.openshift.expose-services=, RELEASE=HEAD, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-14T15:20:42.244 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:41 smithi196 podman[26869]: 2024-04-14 15:20:41.850770753 +0000 UTC m=+0.019034322 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-14T15:20:42.244 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:42 smithi196 podman[26869]: 2024-04-14 15:20:42.143983 +0000 UTC m=+0.312246570 container init 728647a3cffb45d60dafb7e81a8f3fb28ea1b74b6c087e5a28f2668c28778738 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.openshift.expose-services=, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, vendor=Red Hat, Inc., name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, release=754, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-14T15:20:42.244 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:42 smithi196 podman[26869]: 2024-04-14 15:20:42.146446642 +0000 UTC m=+0.314710212 container start 728647a3cffb45d60dafb7e81a8f3fb28ea1b74b6c087e5a28f2668c28778738 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-14T15:20:42.244 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:42 smithi196 ceph-mgr[26888]: -- 172.21.15.196:0/363634049 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e1597d6340 con 0x55e1597af000 2024-04-14T15:20:42.244 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:42 smithi196 bash[26869]: 728647a3cffb45d60dafb7e81a8f3fb28ea1b74b6c087e5a28f2668c28778738 2024-04-14T15:20:42.651 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:42 smithi196 systemd[1]: Started Ceph mgr.x for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:20:42.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:42 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:42.650+0000 7f2067fd9000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:20:42.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:42 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:42.749+0000 7f2067fd9000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:20:43.531 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:43.358+0000 7f2067fd9000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:20:43.531 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:43.447+0000 7f2067fd9000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:20:43.531 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:43.531+0000 7f2067fd9000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:20:43.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-mon[25696]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:43.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:43.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:43.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:43.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:43.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:43.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:43.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:20:43.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:20:43.532 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:43.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[24869]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:43.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:43.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:43.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[29517]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:43.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:20:43.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:20:43.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:43 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-14T15:20:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-14T15:20:43.770 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-14T15:20:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-14T15:20:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00876702 s, 1.2 GB/s 2024-04-14T15:20:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-14T15:20:43.866 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:43.678+0000 7f2067fd9000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:20:43.866 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:43.758+0000 7f2067fd9000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:20:43.866 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:43.865+0000 7f2067fd9000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:20:44.164 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:43.948+0000 7f2067fd9000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:20:44.164 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:44 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:44.110+0000 7f2067fd9000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:20:44.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-04-14T15:20:44.643 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:44 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:44.195+0000 7f2067fd9000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:20:44.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:44 smithi196 ceph-mon[25696]: Reconfiguring mgr.y (unknown last config time)... 2024-04-14T15:20:44.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:44 smithi196 ceph-mon[25696]: Reconfiguring daemon mgr.y on smithi016 2024-04-14T15:20:45.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:44 smithi016 ceph-mon[24869]: Reconfiguring mgr.y (unknown last config time)... 2024-04-14T15:20:45.088 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:44 smithi016 ceph-mon[24869]: Reconfiguring daemon mgr.y on smithi016 2024-04-14T15:20:45.088 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:44 smithi016 ceph-mon[29517]: Reconfiguring mgr.y (unknown last config time)... 2024-04-14T15:20:45.088 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:44 smithi016 ceph-mon[29517]: Reconfiguring daemon mgr.y on smithi016 2024-04-14T15:20:45.540 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:45.221+0000 7f2067fd9000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:20:45.540 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:45.349+0000 7f2067fd9000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:20:45.540 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:45.539+0000 7f2067fd9000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:20:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[24869]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:45.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[29517]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:45.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:45.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:45.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:45.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:45.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:45 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:45.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:45.626+0000 7f2067fd9000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:20:45.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:45.763+0000 7f2067fd9000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:20:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:45 smithi196 ceph-mon[25696]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:45 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:45 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:20:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:45 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:45 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:20:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:45 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:46.382 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:46.081+0000 7f2067fd9000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:20:46.382 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:46.299+0000 7f2067fd9000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:20:46.383 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:46.382+0000 7f2067fd9000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:20:46.647 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:46.551+0000 7f2067fd9000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:20:46.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:20:46.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:20:46.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:46.851 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:20:46.851 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:20:46.851 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:46.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:20:46.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:20:46.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:20:47.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:47 smithi196 ceph-mon[25696]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:47.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:47 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:20:47.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:47 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:20:47.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:47 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:47.757 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:47.465+0000 7f2067fd9000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:20:47.757 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:47.756+0000 7f2067fd9000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:20:47.837 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:47 smithi016 ceph-mon[24869]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:47.837 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:47 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:20:47.837 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:47 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:20:47.837 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:47 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:47.837 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:47 smithi016 ceph-mon[29517]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:47.837 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:47 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:20:47.837 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:47 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:20:47.838 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:47 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:48.164 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:47.849+0000 7f2067fd9000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:20:48.165 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:20:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:20:47.999+0000 7f2067fd9000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:20:48.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:48 smithi196 ceph-mon[25696]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:48.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:48 smithi196 ceph-mon[25696]: Standby manager daemon x started 2024-04-14T15:20:48.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:48 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:20:48.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:48 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:20:48.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:48 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:20:48.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:48 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:20:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[24869]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[24869]: Standby manager daemon x started 2024-04-14T15:20:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:20:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:20:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:20:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:20:49.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[29517]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:20:49.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[29517]: Standby manager daemon x started 2024-04-14T15:20:49.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:20:49.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:20:49.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:20:49.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:48 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/4198935265' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:20:49.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:49 smithi196 ceph-mon[25696]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:49.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:49 smithi196 ceph-mon[25696]: mgrmap e14: y(active, since 63s), standbys: x 2024-04-14T15:20:49.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:20:49.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:49 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/4260705273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8767d4c-2155-4a83-8522-ac3042eaa0be"}]: dispatch 2024-04-14T15:20:49.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:49 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8767d4c-2155-4a83-8522-ac3042eaa0be"}]: dispatch 2024-04-14T15:20:49.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:49 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8767d4c-2155-4a83-8522-ac3042eaa0be"}]': finished 2024-04-14T15:20:49.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:49 smithi196 ceph-mon[25696]: osdmap e5: 1 total, 0 up, 1 in 2024-04-14T15:20:49.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:20:50.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[24869]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[24869]: mgrmap e14: y(active, since 63s), standbys: x 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4260705273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8767d4c-2155-4a83-8522-ac3042eaa0be"}]: dispatch 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8767d4c-2155-4a83-8522-ac3042eaa0be"}]: dispatch 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8767d4c-2155-4a83-8522-ac3042eaa0be"}]': finished 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[24869]: osdmap e5: 1 total, 0 up, 1 in 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[29517]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[29517]: mgrmap e14: y(active, since 63s), standbys: x 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4260705273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8767d4c-2155-4a83-8522-ac3042eaa0be"}]: dispatch 2024-04-14T15:20:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8767d4c-2155-4a83-8522-ac3042eaa0be"}]: dispatch 2024-04-14T15:20:50.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8767d4c-2155-4a83-8522-ac3042eaa0be"}]': finished 2024-04-14T15:20:50.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[29517]: osdmap e5: 1 total, 0 up, 1 in 2024-04-14T15:20:50.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:20:50.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:50 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/639250844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:20:51.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:50 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/639250844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:20:51.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:50 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/639250844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:20:51.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:51 smithi196 ceph-mon[25696]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:52.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:51 smithi016 ceph-mon[24869]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:52.012 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:51 smithi016 ceph-mon[29517]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:53.793 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:53 smithi016 ceph-mon[24869]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:53.794 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:53 smithi016 ceph-mon[29517]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:53.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:53 smithi196 ceph-mon[25696]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:55.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:55 smithi016 ceph-mon[24869]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:55.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:55 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:20:55.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:55 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:55.720 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:55 smithi016 ceph-mon[29517]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:55.720 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:55 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:20:55.720 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:55 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:55 smithi196 ceph-mon[25696]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:55 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:20:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:55 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:20:57.010 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:56 smithi016 ceph-mon[24869]: Deploying daemon osd.0 on smithi016 2024-04-14T15:20:57.010 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:56 smithi016 ceph-mon[29517]: Deploying daemon osd.0 on smithi016 2024-04-14T15:20:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:56 smithi196 ceph-mon[25696]: Deploying daemon osd.0 on smithi016 2024-04-14T15:20:57.786 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:57 smithi016 ceph-mon[24869]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:57.786 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:57 smithi016 ceph-mon[29517]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:58.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:57 smithi196 ceph-mon[25696]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:59.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:20:59 smithi196 ceph-mon[25696]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:59.956 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:20:59 smithi016 ceph-mon[24869]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:20:59.956 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:20:59 smithi016 ceph-mon[29517]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:01.329 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:01 smithi016 ceph-mon[29517]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:01.330 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:01 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:01.330 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:01 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:01.330 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:01 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:01.330 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:01 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:01.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:01 smithi016 ceph-mon[24869]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:01.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:01 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:01.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:01 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:01.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:01 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:01.330 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:01 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:01.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:01 smithi196 ceph-mon[25696]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:01.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:01 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:01.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:01 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:01.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:01 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:01.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:01 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:02.542 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-04-14T15:21:03.711 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.0.service 2024-04-14T15:21:03.713 INFO:tasks.cephadm:Deploying osd.1 on smithi016 with /dev/vg_nvme/lv_3... 2024-04-14T15:21:03.713 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-14T15:21:03.802 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:03 smithi016 ceph-mon[29517]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:03.802 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:03 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:03.802 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:03 smithi016 ceph-mon[29517]: from='osd.0 [v2:172.21.15.16:6802/696303661,v1:172.21.15.16:6803/696303661]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:21:03.802 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:03 smithi016 ceph-mon[29517]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:21:03.803 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:03 smithi016 ceph-mon[24869]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:03.803 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:03 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:03.803 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:03 smithi016 ceph-mon[24869]: from='osd.0 [v2:172.21.15.16:6802/696303661,v1:172.21.15.16:6803/696303661]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:21:03.803 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:03 smithi016 ceph-mon[24869]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:21:03.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:03 smithi196 ceph-mon[25696]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:03.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:03 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:03.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:03 smithi196 ceph-mon[25696]: from='osd.0 [v2:172.21.15.16:6802/696303661,v1:172.21.15.16:6803/696303661]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:21:03.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:03 smithi196 ceph-mon[25696]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:21:04.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[24869]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T15:21:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[24869]: osdmap e6: 1 total, 0 up, 1 in 2024-04-14T15:21:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[24869]: from='osd.0 [v2:172.21.15.16:6802/696303661,v1:172.21.15.16:6803/696303661]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[24869]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[29517]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T15:21:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[29517]: osdmap e6: 1 total, 0 up, 1 in 2024-04-14T15:21:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[29517]: from='osd.0 [v2:172.21.15.16:6802/696303661,v1:172.21.15.16:6803/696303661]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[29517]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:04.612 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[32383]: 2024-04-14T15:21:04.551+0000 7f1f0c508700 -1 osd.0 0 waiting for initial osdmap 2024-04-14T15:21:04.612 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:21:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[32383]: 2024-04-14T15:21:04.554+0000 7f1f0669e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:21:04.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:04 smithi196 ceph-mon[25696]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T15:21:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:04 smithi196 ceph-mon[25696]: osdmap e6: 1 total, 0 up, 1 in 2024-04-14T15:21:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:04 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:04 smithi196 ceph-mon[25696]: from='osd.0 [v2:172.21.15.16:6802/696303661,v1:172.21.15.16:6803/696303661]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:04 smithi196 ceph-mon[25696]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:04 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:05.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:05 smithi016 ceph-mon[29517]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:05.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:05 smithi016 ceph-mon[29517]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:21:05.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:05 smithi016 ceph-mon[29517]: osdmap e7: 1 total, 0 up, 1 in 2024-04-14T15:21:05.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:05 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:05.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:05 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:05.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:05 smithi016 ceph-mon[24869]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:05.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:05 smithi016 ceph-mon[24869]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:21:05.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:05 smithi016 ceph-mon[24869]: osdmap e7: 1 total, 0 up, 1 in 2024-04-14T15:21:05.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:05 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:05.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:05 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:05 smithi196 ceph-mon[25696]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:05 smithi196 ceph-mon[25696]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:21:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:05 smithi196 ceph-mon[25696]: osdmap e7: 1 total, 0 up, 1 in 2024-04-14T15:21:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:05 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:05 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:06 smithi196 ceph-mon[25696]: purged_snaps scrub starts 2024-04-14T15:21:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:06 smithi196 ceph-mon[25696]: purged_snaps scrub ok 2024-04-14T15:21:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:06 smithi196 ceph-mon[25696]: osd.0 [v2:172.21.15.16:6802/696303661,v1:172.21.15.16:6803/696303661] boot 2024-04-14T15:21:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:06 smithi196 ceph-mon[25696]: osdmap e8: 1 total, 1 up, 1 in 2024-04-14T15:21:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:06 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:06.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:06 smithi016 ceph-mon[24869]: purged_snaps scrub starts 2024-04-14T15:21:06.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:06 smithi016 ceph-mon[24869]: purged_snaps scrub ok 2024-04-14T15:21:06.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:06 smithi016 ceph-mon[24869]: osd.0 [v2:172.21.15.16:6802/696303661,v1:172.21.15.16:6803/696303661] boot 2024-04-14T15:21:06.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:06 smithi016 ceph-mon[24869]: osdmap e8: 1 total, 1 up, 1 in 2024-04-14T15:21:06.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:06 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:06.797 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:06 smithi016 ceph-mon[29517]: purged_snaps scrub starts 2024-04-14T15:21:06.797 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:06 smithi016 ceph-mon[29517]: purged_snaps scrub ok 2024-04-14T15:21:06.797 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:06 smithi016 ceph-mon[29517]: osd.0 [v2:172.21.15.16:6802/696303661,v1:172.21.15.16:6803/696303661] boot 2024-04-14T15:21:06.797 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:06 smithi016 ceph-mon[29517]: osdmap e8: 1 total, 1 up, 1 in 2024-04-14T15:21:06.797 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:06 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:21:07.180 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-14T15:21:07.180 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-14T15:21:07.180 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-14T15:21:07.180 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-14T15:21:07.180 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00961404 s, 1.1 GB/s 2024-04-14T15:21:07.180 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-14T15:21:07.834 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:07 smithi016 ceph-mon[29517]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:07.834 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:07 smithi016 ceph-mon[29517]: osdmap e9: 1 total, 1 up, 1 in 2024-04-14T15:21:07.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:07 smithi016 ceph-mon[24869]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:07.834 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:07 smithi016 ceph-mon[24869]: osdmap e9: 1 total, 1 up, 1 in 2024-04-14T15:21:07.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-04-14T15:21:07.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:07 smithi196 ceph-mon[25696]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T15:21:07.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:07 smithi196 ceph-mon[25696]: osdmap e9: 1 total, 1 up, 1 in 2024-04-14T15:21:09.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:09 smithi196 ceph-mon[25696]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:09 smithi196 ceph-mon[25696]: Detected new or changed devices on smithi016 2024-04-14T15:21:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:09 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:09 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:09 smithi196 ceph-mon[25696]: Adjusting osd_memory_target on smithi016 to 16069M 2024-04-14T15:21:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:09 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:09 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:09 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:09 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:09 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:09.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:09 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:09.702 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[29517]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:09.702 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[29517]: Detected new or changed devices on smithi016 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[29517]: Adjusting osd_memory_target on smithi016 to 16069M 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[24869]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[24869]: Detected new or changed devices on smithi016 2024-04-14T15:21:09.703 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:09.704 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:09.704 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[24869]: Adjusting osd_memory_target on smithi016 to 16069M 2024-04-14T15:21:09.704 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:09.704 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:09.704 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:09.704 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:09.704 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:09.704 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:09 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:10.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:10 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:21:10.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:10 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:21:10.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:10 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:10.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:10 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:21:10.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:10 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:21:10.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:10 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:10.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:10 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:21:10.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:10 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:21:10.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:10 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:11.622 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:11 smithi016 ceph-mon[29517]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:11.622 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:11 smithi016 ceph-mon[29517]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:21:11.622 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:11 smithi016 ceph-mon[24869]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:11.622 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:11 smithi016 ceph-mon[24869]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:21:11.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:11 smithi196 ceph-mon[25696]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:11.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:11 smithi196 ceph-mon[25696]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:21:12.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:12 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2279323766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5e2c3d-9d71-41c3-a640-188d71c4fe72"}]: dispatch 2024-04-14T15:21:12.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:12 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5e2c3d-9d71-41c3-a640-188d71c4fe72"}]: dispatch 2024-04-14T15:21:12.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:12 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef5e2c3d-9d71-41c3-a640-188d71c4fe72"}]': finished 2024-04-14T15:21:12.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:12 smithi016 ceph-mon[24869]: osdmap e10: 2 total, 1 up, 2 in 2024-04-14T15:21:12.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:12 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:12.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:12 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2279323766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5e2c3d-9d71-41c3-a640-188d71c4fe72"}]: dispatch 2024-04-14T15:21:12.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:12 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5e2c3d-9d71-41c3-a640-188d71c4fe72"}]: dispatch 2024-04-14T15:21:12.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:12 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef5e2c3d-9d71-41c3-a640-188d71c4fe72"}]': finished 2024-04-14T15:21:12.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:12 smithi016 ceph-mon[29517]: osdmap e10: 2 total, 1 up, 2 in 2024-04-14T15:21:12.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:12 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:12.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:12 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2279323766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5e2c3d-9d71-41c3-a640-188d71c4fe72"}]: dispatch 2024-04-14T15:21:12.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:12 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5e2c3d-9d71-41c3-a640-188d71c4fe72"}]: dispatch 2024-04-14T15:21:12.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:12 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef5e2c3d-9d71-41c3-a640-188d71c4fe72"}]': finished 2024-04-14T15:21:12.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:12 smithi196 ceph-mon[25696]: osdmap e10: 2 total, 1 up, 2 in 2024-04-14T15:21:12.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:12 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:13.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:13 smithi016 ceph-mon[29517]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:13.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:13 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2464498738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:21:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:13 smithi016 ceph-mon[24869]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:13 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2464498738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:21:13.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:13 smithi196 ceph-mon[25696]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:13.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:13 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2464498738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:21:15.828 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:15 smithi016 ceph-mon[29517]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:15.828 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:15 smithi016 ceph-mon[24869]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:15.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:15 smithi196 ceph-mon[25696]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:17 smithi196 ceph-mon[25696]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:18.106 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:17 smithi016 ceph-mon[24869]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:18.107 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:17 smithi016 ceph-mon[29517]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:18.654 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:21:18.654 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:18.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:21:18.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:18.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:21:18.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:19.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:19 smithi196 ceph-mon[25696]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:19 smithi196 ceph-mon[25696]: Deploying daemon osd.1 on smithi016 2024-04-14T15:21:19.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:19 smithi016 ceph-mon[24869]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:19.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:19 smithi016 ceph-mon[24869]: Deploying daemon osd.1 on smithi016 2024-04-14T15:21:19.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:19 smithi016 ceph-mon[29517]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:19.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:19 smithi016 ceph-mon[29517]: Deploying daemon osd.1 on smithi016 2024-04-14T15:21:21.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:21 smithi016 ceph-mon[24869]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:21.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:21 smithi016 ceph-mon[29517]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:21.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:21 smithi196 ceph-mon[25696]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:23.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:23 smithi016 ceph-mon[24869]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:23.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:23 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:23.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:23 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:23.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:23 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:23.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:23 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:23.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:23 smithi016 ceph-mon[29517]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:23.850 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:23 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:23.850 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:23 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:23.850 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:23 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:23.850 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:23 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:23.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:23 smithi196 ceph-mon[25696]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:23.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:23 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:23.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:23 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:23.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:23 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:23.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:23 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:25.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:25 smithi016 ceph-mon[24869]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:25.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:25 smithi016 ceph-mon[29517]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:25.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:25 smithi196 ceph-mon[25696]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:26.634 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 1 on host 'smithi016' 2024-04-14T15:21:26.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:26 smithi016 ceph-mon[24869]: from='osd.1 [v2:172.21.15.16:6810/1936311739,v1:172.21.15.16:6811/1936311739]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:21:26.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:26 smithi016 ceph-mon[24869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:21:26.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:26 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:26.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:26 smithi016 ceph-mon[29517]: from='osd.1 [v2:172.21.15.16:6810/1936311739,v1:172.21.15.16:6811/1936311739]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:21:26.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:26 smithi016 ceph-mon[29517]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:21:26.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:26 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:27.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:26 smithi196 ceph-mon[25696]: from='osd.1 [v2:172.21.15.16:6810/1936311739,v1:172.21.15.16:6811/1936311739]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:21:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:26 smithi196 ceph-mon[25696]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:21:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:26 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:27.198 DEBUG:teuthology.orchestra.run.smithi016:osd.1> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.1.service 2024-04-14T15:21:27.201 INFO:tasks.cephadm:Deploying osd.2 on smithi016 with /dev/vg_nvme/lv_2... 2024-04-14T15:21:27.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-14T15:21:27.860 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[35336]: 2024-04-14T15:21:27.674+0000 7fe5fac6c700 -1 osd.1 0 waiting for initial osdmap 2024-04-14T15:21:27.861 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[35336]: 2024-04-14T15:21:27.677+0000 7fe5f5e04700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:21:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[24869]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[24869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T15:21:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[24869]: osdmap e11: 2 total, 1 up, 2 in 2024-04-14T15:21:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[24869]: from='osd.1 [v2:172.21.15.16:6810/1936311739,v1:172.21.15.16:6811/1936311739]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[24869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:27.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[29517]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:27.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[29517]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T15:21:27.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[29517]: osdmap e11: 2 total, 1 up, 2 in 2024-04-14T15:21:27.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:27.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[29517]: from='osd.1 [v2:172.21.15.16:6810/1936311739,v1:172.21.15.16:6811/1936311739]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:27.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[29517]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:27.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:27 smithi196 ceph-mon[25696]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:27 smithi196 ceph-mon[25696]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T15:21:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:27 smithi196 ceph-mon[25696]: osdmap e11: 2 total, 1 up, 2 in 2024-04-14T15:21:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:27 smithi196 ceph-mon[25696]: from='osd.1 [v2:172.21.15.16:6810/1936311739,v1:172.21.15.16:6811/1936311739]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:27 smithi196 ceph-mon[25696]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:28.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:28 smithi016 ceph-mon[24869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:21:28.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:28 smithi016 ceph-mon[24869]: osdmap e12: 2 total, 1 up, 2 in 2024-04-14T15:21:28.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:28 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:28.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:28 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:28.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:28 smithi016 ceph-mon[29517]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:21:28.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:28 smithi016 ceph-mon[29517]: osdmap e12: 2 total, 1 up, 2 in 2024-04-14T15:21:28.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:28 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:28.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:28 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:29.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:28 smithi196 ceph-mon[25696]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:21:29.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:28 smithi196 ceph-mon[25696]: osdmap e12: 2 total, 1 up, 2 in 2024-04-14T15:21:29.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:28 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:29.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:28 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:29.305 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-14T15:21:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-14T15:21:29.306 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-14T15:21:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-14T15:21:29.306 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00929585 s, 1.1 GB/s 2024-04-14T15:21:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-14T15:21:29.961 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[29517]: purged_snaps scrub starts 2024-04-14T15:21:29.961 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[29517]: purged_snaps scrub ok 2024-04-14T15:21:29.961 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[29517]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:29.961 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:29.961 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[29517]: osd.1 [v2:172.21.15.16:6810/1936311739,v1:172.21.15.16:6811/1936311739] boot 2024-04-14T15:21:29.961 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[29517]: osdmap e13: 2 total, 2 up, 2 in 2024-04-14T15:21:29.961 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:29.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[24869]: purged_snaps scrub starts 2024-04-14T15:21:29.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[24869]: purged_snaps scrub ok 2024-04-14T15:21:29.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[24869]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:29.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:29.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[24869]: osd.1 [v2:172.21.15.16:6810/1936311739,v1:172.21.15.16:6811/1936311739] boot 2024-04-14T15:21:29.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[24869]: osdmap e13: 2 total, 2 up, 2 in 2024-04-14T15:21:29.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:29 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:29.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-04-14T15:21:30.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:29 smithi196 ceph-mon[25696]: purged_snaps scrub starts 2024-04-14T15:21:30.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:29 smithi196 ceph-mon[25696]: purged_snaps scrub ok 2024-04-14T15:21:30.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:29 smithi196 ceph-mon[25696]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T15:21:30.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:29 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:30.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:29 smithi196 ceph-mon[25696]: osd.1 [v2:172.21.15.16:6810/1936311739,v1:172.21.15.16:6811/1936311739] boot 2024-04-14T15:21:30.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:29 smithi196 ceph-mon[25696]: osdmap e13: 2 total, 2 up, 2 in 2024-04-14T15:21:30.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:29 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:21:30.934 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:30 smithi016 ceph-mon[29517]: osdmap e14: 2 total, 2 up, 2 in 2024-04-14T15:21:30.935 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:30 smithi016 ceph-mon[29517]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:30.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:30 smithi016 ceph-mon[24869]: osdmap e14: 2 total, 2 up, 2 in 2024-04-14T15:21:30.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:30 smithi016 ceph-mon[24869]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:31.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:30 smithi196 ceph-mon[25696]: osdmap e14: 2 total, 2 up, 2 in 2024-04-14T15:21:31.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:30 smithi196 ceph-mon[25696]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:33.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[29517]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:33.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:33.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:33.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:33.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:33.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:33.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:33.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:33.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:33.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:33.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[24869]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:33.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:33.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:33.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:33.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:33.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:33.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:33.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:33.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:33.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:32 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:33.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:32 smithi196 ceph-mon[25696]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:32 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:32 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:32 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:32 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:32 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:32 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:32 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:32 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:32 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:34.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[24869]: Detected new or changed devices on smithi016 2024-04-14T15:21:34.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[24869]: Adjusting osd_memory_target on smithi016 to 8034M 2024-04-14T15:21:34.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[24869]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:21:34.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:21:34.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:21:34.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:34.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[29517]: Detected new or changed devices on smithi016 2024-04-14T15:21:34.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[29517]: Adjusting osd_memory_target on smithi016 to 8034M 2024-04-14T15:21:34.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[29517]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:21:34.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:21:34.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:21:34.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:33 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:34.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:33 smithi196 ceph-mon[25696]: Detected new or changed devices on smithi016 2024-04-14T15:21:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:33 smithi196 ceph-mon[25696]: Adjusting osd_memory_target on smithi016 to 8034M 2024-04-14T15:21:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:33 smithi196 ceph-mon[25696]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:21:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:21:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:21:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:33 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:35 smithi196 ceph-mon[25696]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:35 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/3342806771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1e7e85-29f9-4b4f-a16c-d80ed36951c1"}]: dispatch 2024-04-14T15:21:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:35 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1e7e85-29f9-4b4f-a16c-d80ed36951c1"}]: dispatch 2024-04-14T15:21:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:35 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b1e7e85-29f9-4b4f-a16c-d80ed36951c1"}]': finished 2024-04-14T15:21:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:35 smithi196 ceph-mon[25696]: osdmap e15: 3 total, 2 up, 3 in 2024-04-14T15:21:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:35 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[24869]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3342806771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1e7e85-29f9-4b4f-a16c-d80ed36951c1"}]: dispatch 2024-04-14T15:21:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1e7e85-29f9-4b4f-a16c-d80ed36951c1"}]: dispatch 2024-04-14T15:21:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b1e7e85-29f9-4b4f-a16c-d80ed36951c1"}]': finished 2024-04-14T15:21:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[24869]: osdmap e15: 3 total, 2 up, 3 in 2024-04-14T15:21:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[29517]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3342806771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1e7e85-29f9-4b4f-a16c-d80ed36951c1"}]: dispatch 2024-04-14T15:21:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1e7e85-29f9-4b4f-a16c-d80ed36951c1"}]: dispatch 2024-04-14T15:21:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b1e7e85-29f9-4b4f-a16c-d80ed36951c1"}]': finished 2024-04-14T15:21:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[29517]: osdmap e15: 3 total, 2 up, 3 in 2024-04-14T15:21:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:35 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:36.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:36 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1065370671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:21:37.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:36 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1065370671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:21:37.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:36 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1065370671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:21:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:37 smithi196 ceph-mon[25696]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:38.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:37 smithi016 ceph-mon[24869]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:38.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:37 smithi016 ceph-mon[29517]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:39 smithi016 ceph-mon[24869]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:39.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:39 smithi016 ceph-mon[29517]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:39.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:39 smithi196 ceph-mon[25696]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:41.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:41 smithi196 ceph-mon[25696]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:42.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:41 smithi016 ceph-mon[24869]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:42.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:41 smithi016 ceph-mon[29517]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:42.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:42 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:21:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:42 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:42 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:21:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:42 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:43.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:42 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:21:43.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:42 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:43.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:43 smithi016 ceph-mon[24869]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:43.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:43 smithi016 ceph-mon[24869]: Deploying daemon osd.2 on smithi016 2024-04-14T15:21:43.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:43 smithi016 ceph-mon[29517]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:43.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:43 smithi016 ceph-mon[29517]: Deploying daemon osd.2 on smithi016 2024-04-14T15:21:44.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:43 smithi196 ceph-mon[25696]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:44.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:43 smithi196 ceph-mon[25696]: Deploying daemon osd.2 on smithi016 2024-04-14T15:21:46.005 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:45 smithi016 ceph-mon[24869]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:46.005 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:45 smithi016 ceph-mon[29517]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:46.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:45 smithi196 ceph-mon[25696]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:46.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:21:46.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:21:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:46.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:21:46.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:21:46.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:46.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:46.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:46.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:47.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:21:47.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:21:47.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:47.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:47.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:47.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:47.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:47 smithi016 ceph-mon[29517]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:47.939 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:47 smithi016 ceph-mon[24869]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:48.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:47 smithi196 ceph-mon[25696]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:49.603 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 2 on host 'smithi016' 2024-04-14T15:21:49.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:49 smithi016 ceph-mon[29517]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:49.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:49.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:49 smithi016 ceph-mon[24869]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:49.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:49.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:49 smithi196 ceph-mon[25696]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:49.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:50.224 DEBUG:teuthology.orchestra.run.smithi016:osd.2> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.2.service 2024-04-14T15:21:50.227 INFO:tasks.cephadm:Deploying osd.3 on smithi016 with /dev/vg_nvme/lv_1... 2024-04-14T15:21:50.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-14T15:21:50.839 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:50 smithi016 ceph-mon[24869]: from='osd.2 [v2:172.21.15.16:6818/701104650,v1:172.21.15.16:6819/701104650]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:21:50.839 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:50 smithi016 ceph-mon[24869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:21:50.839 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:50 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:50.840 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:50 smithi016 ceph-mon[29517]: from='osd.2 [v2:172.21.15.16:6818/701104650,v1:172.21.15.16:6819/701104650]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:21:50.840 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:50 smithi016 ceph-mon[29517]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:21:50.840 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:50 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:50.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:50 smithi196 ceph-mon[25696]: from='osd.2 [v2:172.21.15.16:6818/701104650,v1:172.21.15.16:6819/701104650]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:21:50.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:50 smithi196 ceph-mon[25696]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:21:50.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:50 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:51.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:51 smithi196 ceph-mon[25696]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:51.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:51 smithi196 ceph-mon[25696]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T15:21:51.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:51 smithi196 ceph-mon[25696]: osdmap e16: 3 total, 2 up, 3 in 2024-04-14T15:21:51.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:51 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:51.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:51 smithi196 ceph-mon[25696]: from='osd.2 [v2:172.21.15.16:6818/701104650,v1:172.21.15.16:6819/701104650]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:51.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:51 smithi196 ceph-mon[25696]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:51.965 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[24869]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:51.965 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[24869]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T15:21:51.965 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[24869]: osdmap e16: 3 total, 2 up, 3 in 2024-04-14T15:21:51.965 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:51.965 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[24869]: from='osd.2 [v2:172.21.15.16:6818/701104650,v1:172.21.15.16:6819/701104650]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:51.965 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[24869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:51.966 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[29517]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:51.966 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[29517]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T15:21:51.966 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[29517]: osdmap e16: 3 total, 2 up, 3 in 2024-04-14T15:21:51.966 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:51.966 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[29517]: from='osd.2 [v2:172.21.15.16:6818/701104650,v1:172.21.15.16:6819/701104650]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:51.966 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-mon[29517]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:21:51.966 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[38251]: 2024-04-14T15:21:51.667+0000 7f1ca7bff700 -1 osd.2 0 waiting for initial osdmap 2024-04-14T15:21:51.966 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:21:51 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[38251]: 2024-04-14T15:21:51.671+0000 7f1ca3598700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:21:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-14T15:21:52.440 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-14T15:21:52.440 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-04-14T15:21:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-04-14T15:21:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00975159 s, 1.1 GB/s 2024-04-14T15:21:52.441 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-04-14T15:21:52.828 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[29517]: purged_snaps scrub starts 2024-04-14T15:21:52.828 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[29517]: purged_snaps scrub ok 2024-04-14T15:21:52.828 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[29517]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:21:52.828 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[29517]: osdmap e17: 3 total, 2 up, 3 in 2024-04-14T15:21:52.829 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:52.829 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:52.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[24869]: purged_snaps scrub starts 2024-04-14T15:21:52.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[24869]: purged_snaps scrub ok 2024-04-14T15:21:52.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[24869]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:21:52.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[24869]: osdmap e17: 3 total, 2 up, 3 in 2024-04-14T15:21:52.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:52.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:52 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:53.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2024-04-14T15:21:53.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:52 smithi196 ceph-mon[25696]: purged_snaps scrub starts 2024-04-14T15:21:53.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:52 smithi196 ceph-mon[25696]: purged_snaps scrub ok 2024-04-14T15:21:53.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:52 smithi196 ceph-mon[25696]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:21:53.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:52 smithi196 ceph-mon[25696]: osdmap e17: 3 total, 2 up, 3 in 2024-04-14T15:21:53.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:52 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:53.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:52 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:53.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:53 smithi016 ceph-mon[24869]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:53.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:53 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:53.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:53 smithi016 ceph-mon[24869]: osd.2 [v2:172.21.15.16:6818/701104650,v1:172.21.15.16:6819/701104650] boot 2024-04-14T15:21:53.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:53 smithi016 ceph-mon[24869]: osdmap e18: 3 total, 3 up, 3 in 2024-04-14T15:21:53.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:53 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:53.716 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:53 smithi016 ceph-mon[29517]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:53.716 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:53 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:53.716 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:53 smithi016 ceph-mon[29517]: osd.2 [v2:172.21.15.16:6818/701104650,v1:172.21.15.16:6819/701104650] boot 2024-04-14T15:21:53.717 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:53 smithi016 ceph-mon[29517]: osdmap e18: 3 total, 3 up, 3 in 2024-04-14T15:21:53.717 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:53 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:54.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:53 smithi196 ceph-mon[25696]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T15:21:54.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:53 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:54.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:53 smithi196 ceph-mon[25696]: osd.2 [v2:172.21.15.16:6818/701104650,v1:172.21.15.16:6819/701104650] boot 2024-04-14T15:21:54.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:53 smithi196 ceph-mon[25696]: osdmap e18: 3 total, 3 up, 3 in 2024-04-14T15:21:54.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:53 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:21:54.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:54 smithi016 ceph-mon[24869]: osdmap e19: 3 total, 3 up, 3 in 2024-04-14T15:21:54.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:54 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-14T15:21:54.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:54 smithi016 ceph-mon[29517]: osdmap e19: 3 total, 3 up, 3 in 2024-04-14T15:21:54.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:54 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-14T15:21:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:54 smithi196 ceph-mon[25696]: osdmap e19: 3 total, 3 up, 3 in 2024-04-14T15:21:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:54 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-14T15:21:55.796 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:55 smithi016 ceph-mon[29517]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:21:55.797 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:55 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-14T15:21:55.797 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:55 smithi016 ceph-mon[29517]: osdmap e20: 3 total, 3 up, 3 in 2024-04-14T15:21:55.797 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:55 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:55.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:55 smithi016 ceph-mon[24869]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:21:55.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:55 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-14T15:21:55.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:55 smithi016 ceph-mon[24869]: osdmap e20: 3 total, 3 up, 3 in 2024-04-14T15:21:55.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:55 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:56.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:55 smithi196 ceph-mon[25696]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:21:56.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:55 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-14T15:21:56.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:55 smithi196 ceph-mon[25696]: osdmap e20: 3 total, 3 up, 3 in 2024-04-14T15:21:56.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:55 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T15:21:56.596 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:21:56 smithi016 sudo[39594]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:21:56.596 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:21:56 smithi016 sudo[39594]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:56.596 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:21:56 smithi016 sudo[39594]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:56.850 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:21:56 smithi016 sudo[39658]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:21:56.850 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:21:56 smithi016 sudo[39658]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:56.851 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:21:56 smithi016 sudo[39658]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:56.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:56.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: osdmap e21: 3 total, 3 up, 3 in 2024-04-14T15:21:56.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:21:56.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: Detected new or changed devices on smithi016 2024-04-14T15:21:56.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:56.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: Adjusting osd_memory_target on smithi016 to 5356M 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:56.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: osdmap e21: 3 total, 3 up, 3 in 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: Detected new or changed devices on smithi016 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: Adjusting osd_memory_target on smithi016 to 5356M 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:56.853 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:56.854 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:56.854 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:21:56.854 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:21:56.854 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:21:56.854 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:56.854 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:56 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:57.110 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:21:56 smithi016 sudo[39665]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:21:57.110 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:21:56 smithi016 sudo[39665]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:57.110 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:21:56 smithi016 sudo[39665]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: osdmap e21: 3 total, 3 up, 3 in 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: Detected new or changed devices on smithi016 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: Adjusting osd_memory_target on smithi016 to 5356M 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:57.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:21:57.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:57.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:21:57.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:57.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:21:57.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:21:57.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:21:57.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:21:57.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:56 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:21:57.396 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39681]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:21:57.396 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39681]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:57.396 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39681]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:57.661 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39687]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:21:57.661 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39687]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:57.661 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39687]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:57.923 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39691]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:21:57.924 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39691]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:57.924 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39691]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:58.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39717]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:21:58.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39717]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:58.187 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:57 smithi016 sudo[39717]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:58.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:58 smithi016 ceph-mon[29517]: osdmap e22: 3 total, 3 up, 3 in 2024-04-14T15:21:58.448 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:58 smithi016 ceph-mon[29517]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:21:58.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:58 smithi016 sudo[39723]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:21:58.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:58 smithi016 sudo[39723]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:58.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:58 smithi016 sudo[39723]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:58.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:58 smithi016 ceph-mon[24869]: osdmap e22: 3 total, 3 up, 3 in 2024-04-14T15:21:58.448 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:58 smithi016 ceph-mon[24869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:21:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:58 smithi196 ceph-mon[25696]: osdmap e22: 3 total, 3 up, 3 in 2024-04-14T15:21:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:58 smithi196 ceph-mon[25696]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:21:58.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:58 smithi016 sudo[39739]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:21:58.719 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:58 smithi016 sudo[39739]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:58.719 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:58 smithi016 sudo[39739]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:59.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:58 smithi016 sudo[39762]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:21:59.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:58 smithi016 sudo[39762]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:59.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:58 smithi016 sudo[39762]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:59.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:58 smithi196 sudo[27214]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T15:21:59.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:58 smithi196 sudo[27214]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:59.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:58 smithi196 sudo[27214]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:21:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:21:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:21:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:21:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:21:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:21:59.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2169698958' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f943ae4-952d-4e12-97c5-2a23c47a1c42"}]: dispatch 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2169698958' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f943ae4-952d-4e12-97c5-2a23c47a1c42"}]': finished 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: osdmap e23: 4 total, 3 up, 4 in 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:21:59.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2169698958' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f943ae4-952d-4e12-97c5-2a23c47a1c42"}]: dispatch 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2169698958' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f943ae4-952d-4e12-97c5-2a23c47a1c42"}]': finished 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: osdmap e23: 4 total, 3 up, 4 in 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:21:59.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:21:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:21:59.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 sudo[27217]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 sudo[27217]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 sudo[27217]: pam_unix(sudo:session): session closed for user root 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:21:59.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2169698958' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f943ae4-952d-4e12-97c5-2a23c47a1c42"}]: dispatch 2024-04-14T15:21:59.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:21:59.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:21:59.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:21:59.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2169698958' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f943ae4-952d-4e12-97c5-2a23c47a1c42"}]': finished 2024-04-14T15:21:59.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: osdmap e23: 4 total, 3 up, 4 in 2024-04-14T15:21:59.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:21:59.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:21:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:00.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:00 smithi016 ceph-mon[24869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:22:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:00 smithi016 ceph-mon[24869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:22:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:00 smithi016 ceph-mon[24869]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-14T15:22:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:00 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:00 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/850717873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:22:00.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:00 smithi016 ceph-mon[29517]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:22:00.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:00 smithi016 ceph-mon[29517]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:22:00.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:00 smithi016 ceph-mon[29517]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-14T15:22:00.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:00 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:00.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:00 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/850717873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:22:00.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:00 smithi196 ceph-mon[25696]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T15:22:00.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:00 smithi196 ceph-mon[25696]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T15:22:00.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:00 smithi196 ceph-mon[25696]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-14T15:22:00.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:00 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:00.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:00 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/850717873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:22:01.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:01 smithi016 ceph-mon[24869]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:01.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:01 smithi016 ceph-mon[29517]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:01.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:01 smithi196 ceph-mon[25696]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:03.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:03 smithi016 ceph-mon[24869]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:03.512 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:03 smithi016 ceph-mon[29517]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:03.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:03 smithi196 ceph-mon[25696]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:05 smithi196 ceph-mon[25696]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:05 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:22:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:05 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:05.956 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:05 smithi016 ceph-mon[24869]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:05.956 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:05 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:22:05.956 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:05 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:05.956 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:05 smithi016 ceph-mon[29517]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:05.956 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:05 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:22:05.956 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:05 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:06.989 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:06 smithi016 ceph-mon[29517]: Deploying daemon osd.3 on smithi016 2024-04-14T15:22:06.989 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:06 smithi016 ceph-mon[24869]: Deploying daemon osd.3 on smithi016 2024-04-14T15:22:07.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:06 smithi196 ceph-mon[25696]: Deploying daemon osd.3 on smithi016 2024-04-14T15:22:07.818 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:07 smithi016 ceph-mon[29517]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:07.819 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:07 smithi016 ceph-mon[24869]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:08.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:07 smithi196 ceph-mon[25696]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:09.933 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:09 smithi016 ceph-mon[24869]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:09.933 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:09 smithi016 ceph-mon[29517]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:10.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:09 smithi196 ceph-mon[25696]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:11.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:11 smithi016 ceph-mon[24869]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:11.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:11 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:11.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:11 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:11.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:11 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:11.576 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:11 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:11.577 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:11 smithi016 ceph-mon[29517]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:11.577 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:11 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:11.577 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:11 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:11.577 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:11 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:11.577 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:11 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:11.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:11 smithi196 ceph-mon[25696]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:11.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:11 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:11.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:11 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:11.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:11 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:11.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:11 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:13.334 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 3 on host 'smithi016' 2024-04-14T15:22:13.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:13 smithi016 ceph-mon[24869]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:13.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:13 smithi016 ceph-mon[29517]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:13.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:13 smithi196 ceph-mon[25696]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:13.952 DEBUG:teuthology.orchestra.run.smithi016:osd.3> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.3.service 2024-04-14T15:22:13.954 INFO:tasks.cephadm:Deploying osd.4 on smithi196 with /dev/vg_nvme/lv_4... 2024-04-14T15:22:13.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-14T15:22:14.548 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:14 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:14.548 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:14 smithi196 ceph-mon[25696]: from='osd.3 [v2:172.21.15.16:6826/3777247940,v1:172.21.15.16:6827/3777247940]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:22:14.548 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:14 smithi196 ceph-mon[25696]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:22:14.548 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:14 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:14.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:14 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:14.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:14 smithi016 ceph-mon[29517]: from='osd.3 [v2:172.21.15.16:6826/3777247940,v1:172.21.15.16:6827/3777247940]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:22:14.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:14 smithi016 ceph-mon[29517]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:22:14.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:14 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:14.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:14 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:14.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:14 smithi016 ceph-mon[24869]: from='osd.3 [v2:172.21.15.16:6826/3777247940,v1:172.21.15.16:6827/3777247940]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:22:14.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:14 smithi016 ceph-mon[24869]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:22:14.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:14 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:15.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[29517]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:15.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[29517]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T15:22:15.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[29517]: osdmap e24: 4 total, 3 up, 4 in 2024-04-14T15:22:15.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:15.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[29517]: from='osd.3 [v2:172.21.15.16:6826/3777247940,v1:172.21.15.16:6827/3777247940]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:22:15.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[29517]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:22:15.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[24869]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:15.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[24869]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T15:22:15.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[24869]: osdmap e24: 4 total, 3 up, 4 in 2024-04-14T15:22:15.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:15.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[24869]: from='osd.3 [v2:172.21.15.16:6826/3777247940,v1:172.21.15.16:6827/3777247940]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:22:15.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-mon[24869]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:22:15.362 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[41211]: 2024-04-14T15:22:15.343+0000 7fd4184d0700 -1 osd.3 0 waiting for initial osdmap 2024-04-14T15:22:15.362 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:22:15 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[41211]: 2024-04-14T15:22:15.347+0000 7fd412e67700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:22:15.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:15 smithi196 ceph-mon[25696]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:15.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:15 smithi196 ceph-mon[25696]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T15:22:15.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:15 smithi196 ceph-mon[25696]: osdmap e24: 4 total, 3 up, 4 in 2024-04-14T15:22:15.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:15 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:15.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:15 smithi196 ceph-mon[25696]: from='osd.3 [v2:172.21.15.16:6826/3777247940,v1:172.21.15.16:6827/3777247940]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:22:15.604 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:15 smithi196 ceph-mon[25696]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:22:15.919 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-14T15:22:15.919 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-14T15:22:15.919 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-04-14T15:22:15.919 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-04-14T15:22:15.919 INFO:teuthology.orchestra.run.smithi196.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0139695 s, 751 MB/s 2024-04-14T15:22:15.919 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-04-14T15:22:16.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:16 smithi016 ceph-mon[24869]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:22:16.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:16 smithi016 ceph-mon[24869]: osdmap e25: 4 total, 3 up, 4 in 2024-04-14T15:22:16.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:16 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:16.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:16 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:16.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:16 smithi016 ceph-mon[29517]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:22:16.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:16 smithi016 ceph-mon[29517]: osdmap e25: 4 total, 3 up, 4 in 2024-04-14T15:22:16.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:16.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:16 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:16.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_4 2024-04-14T15:22:16.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:16 smithi196 ceph-mon[25696]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-04-14T15:22:16.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:16 smithi196 ceph-mon[25696]: osdmap e25: 4 total, 3 up, 4 in 2024-04-14T15:22:16.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:16 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:16.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:16 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:17.538 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[24869]: purged_snaps scrub starts 2024-04-14T15:22:17.538 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[24869]: purged_snaps scrub ok 2024-04-14T15:22:17.538 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[24869]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:17.538 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[24869]: osd.3 [v2:172.21.15.16:6826/3777247940,v1:172.21.15.16:6827/3777247940] boot 2024-04-14T15:22:17.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[24869]: osdmap e26: 4 total, 4 up, 4 in 2024-04-14T15:22:17.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:17.539 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[29517]: purged_snaps scrub starts 2024-04-14T15:22:17.539 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[29517]: purged_snaps scrub ok 2024-04-14T15:22:17.539 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[29517]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:17.539 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[29517]: osd.3 [v2:172.21.15.16:6826/3777247940,v1:172.21.15.16:6827/3777247940] boot 2024-04-14T15:22:17.539 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[29517]: osdmap e26: 4 total, 4 up, 4 in 2024-04-14T15:22:17.539 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:17 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:17.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:17 smithi196 ceph-mon[25696]: purged_snaps scrub starts 2024-04-14T15:22:17.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:17 smithi196 ceph-mon[25696]: purged_snaps scrub ok 2024-04-14T15:22:17.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:17 smithi196 ceph-mon[25696]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-14T15:22:17.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:17 smithi196 ceph-mon[25696]: osd.3 [v2:172.21.15.16:6826/3777247940,v1:172.21.15.16:6827/3777247940] boot 2024-04-14T15:22:17.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:17 smithi196 ceph-mon[25696]: osdmap e26: 4 total, 4 up, 4 in 2024-04-14T15:22:17.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:17 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: osdmap e27: 4 total, 4 up, 4 in 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: Detected new or changed devices on smithi016 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: Adjusting osd_memory_target on smithi016 to 4017M 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: osdmap e27: 4 total, 4 up, 4 in 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: Detected new or changed devices on smithi016 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: Adjusting osd_memory_target on smithi016 to 4017M 2024-04-14T15:22:18.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:18.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:18.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:18.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:22:18.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:22:18.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:18 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:18.654 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: osdmap e27: 4 total, 4 up, 4 in 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: Detected new or changed devices on smithi016 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: Adjusting osd_memory_target on smithi016 to 4017M 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:22:18.655 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:22:18.656 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:18 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:19 smithi196 ceph-mon[25696]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:19 smithi196 ceph-mon[25696]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:22:20.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:19 smithi016 ceph-mon[29517]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:20.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:19 smithi016 ceph-mon[29517]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:22:20.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:19 smithi016 ceph-mon[24869]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:20.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:19 smithi016 ceph-mon[24869]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:22:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:20 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/710480567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ad25b86-55ee-4769-991e-de2a723b3516"}]: dispatch 2024-04-14T15:22:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:20 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ad25b86-55ee-4769-991e-de2a723b3516"}]: dispatch 2024-04-14T15:22:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:20 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ad25b86-55ee-4769-991e-de2a723b3516"}]': finished 2024-04-14T15:22:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:20 smithi016 ceph-mon[24869]: osdmap e28: 5 total, 4 up, 5 in 2024-04-14T15:22:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:20 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:20 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/710480567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ad25b86-55ee-4769-991e-de2a723b3516"}]: dispatch 2024-04-14T15:22:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:20 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ad25b86-55ee-4769-991e-de2a723b3516"}]: dispatch 2024-04-14T15:22:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:20 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ad25b86-55ee-4769-991e-de2a723b3516"}]': finished 2024-04-14T15:22:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:20 smithi016 ceph-mon[29517]: osdmap e28: 5 total, 4 up, 5 in 2024-04-14T15:22:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:20 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:21.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:20 smithi196 ceph-mon[25696]: from='client.? 172.21.15.196:0/710480567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ad25b86-55ee-4769-991e-de2a723b3516"}]: dispatch 2024-04-14T15:22:21.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:20 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ad25b86-55ee-4769-991e-de2a723b3516"}]: dispatch 2024-04-14T15:22:21.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:20 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ad25b86-55ee-4769-991e-de2a723b3516"}]': finished 2024-04-14T15:22:21.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:20 smithi196 ceph-mon[25696]: osdmap e28: 5 total, 4 up, 5 in 2024-04-14T15:22:21.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:20 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:22.012 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:21 smithi016 ceph-mon[29517]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:22.012 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:21 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/3442524839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:22:22.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:21 smithi016 ceph-mon[24869]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:22.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:21 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/3442524839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:22:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:21 smithi196 ceph-mon[25696]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:21 smithi196 ceph-mon[25696]: from='client.? 172.21.15.196:0/3442524839' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:22:23.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:23 smithi196 ceph-mon[25696]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:24.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:23 smithi016 ceph-mon[24869]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:24.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:23 smithi016 ceph-mon[29517]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:25.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:25 smithi196 ceph-mon[25696]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:26.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:25 smithi016 ceph-mon[24869]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:26.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:25 smithi016 ceph-mon[29517]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:26.946 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:26 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:22:26.947 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:26 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:27.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:26 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:22:27.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:26 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:26 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:22:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:26 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:27.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:27 smithi196 ceph-mon[25696]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:27.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:27 smithi196 ceph-mon[25696]: Deploying daemon osd.4 on smithi196 2024-04-14T15:22:28.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:27 smithi016 ceph-mon[24869]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:28.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:27 smithi016 ceph-mon[24869]: Deploying daemon osd.4 on smithi196 2024-04-14T15:22:28.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:27 smithi016 ceph-mon[29517]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:28.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:27 smithi016 ceph-mon[29517]: Deploying daemon osd.4 on smithi196 2024-04-14T15:22:29.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:29 smithi196 ceph-mon[25696]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:30.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:29 smithi016 ceph-mon[24869]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:30.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:29 smithi016 ceph-mon[29517]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:31.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:31 smithi196 ceph-mon[25696]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:31.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:31 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:31.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:31 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:31.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:31 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:31.845 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:31 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:32.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:31 smithi016 ceph-mon[24869]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:32.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:32.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:32.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:32.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:31 smithi016 ceph-mon[29517]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:32.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:33.020 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:32 smithi196 ceph-mon[25696]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:33.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:32 smithi016 ceph-mon[24869]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:33.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:32 smithi016 ceph-mon[29517]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:34.548 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 4 on host 'smithi196' 2024-04-14T15:22:34.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:34 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:34.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:34 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:34.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:34 smithi016 ceph-mon[29517]: from='osd.4 [v2:172.21.15.196:6800/61582609,v1:172.21.15.196:6801/61582609]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:22:34.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:34 smithi016 ceph-mon[29517]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:22:34.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:34 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:34.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:34 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:34.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:34 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:34.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:34 smithi016 ceph-mon[24869]: from='osd.4 [v2:172.21.15.196:6800/61582609,v1:172.21.15.196:6801/61582609]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:22:34.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:34 smithi016 ceph-mon[24869]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:22:34.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:34 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:34.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:34 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:34 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:34 smithi196 ceph-mon[25696]: from='osd.4 [v2:172.21.15.196:6800/61582609,v1:172.21.15.196:6801/61582609]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:22:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:34 smithi196 ceph-mon[25696]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:22:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:34 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:35.234 DEBUG:teuthology.orchestra.run.smithi196:osd.4> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.4.service 2024-04-14T15:22:35.236 INFO:tasks.cephadm:Deploying osd.5 on smithi196 with /dev/vg_nvme/lv_3... 2024-04-14T15:22:35.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-14T15:22:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[29517]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[29517]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T15:22:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[29517]: osdmap e29: 5 total, 4 up, 5 in 2024-04-14T15:22:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[29517]: from='osd.4 [v2:172.21.15.196:6800/61582609,v1:172.21.15.196:6801/61582609]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:22:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[29517]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:22:35.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[24869]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:35.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:35.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:35.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:35.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[24869]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T15:22:35.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[24869]: osdmap e29: 5 total, 4 up, 5 in 2024-04-14T15:22:35.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:35.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[24869]: from='osd.4 [v2:172.21.15.196:6800/61582609,v1:172.21.15.196:6801/61582609]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:22:35.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:35 smithi016 ceph-mon[24869]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:22:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:35 smithi196 ceph-mon[25696]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:35 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:35 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:35 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:35 smithi196 ceph-mon[25696]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T15:22:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:35 smithi196 ceph-mon[25696]: osdmap e29: 5 total, 4 up, 5 in 2024-04-14T15:22:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:35 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:35 smithi196 ceph-mon[25696]: from='osd.4 [v2:172.21.15.196:6800/61582609,v1:172.21.15.196:6801/61582609]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:22:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:35 smithi196 ceph-mon[25696]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:22:35.916 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:22:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[29135]: 2024-04-14T15:22:35.674+0000 7ffba5fd6700 -1 osd.4 0 waiting for initial osdmap 2024-04-14T15:22:35.916 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:22:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[29135]: 2024-04-14T15:22:35.678+0000 7ffba096d700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:22:36.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:36 smithi196 ceph-mon[25696]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:22:36.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:36 smithi196 ceph-mon[25696]: osdmap e30: 5 total, 4 up, 5 in 2024-04-14T15:22:36.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:36 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:36.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:36 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:37.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:36 smithi016 ceph-mon[29517]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:22:37.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:36 smithi016 ceph-mon[29517]: osdmap e30: 5 total, 4 up, 5 in 2024-04-14T15:22:37.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:36 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:37.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:36 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:37.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:36 smithi016 ceph-mon[24869]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:22:37.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:36 smithi016 ceph-mon[24869]: osdmap e30: 5 total, 4 up, 5 in 2024-04-14T15:22:37.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:36 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:37.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:36 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:37.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:37 smithi196 ceph-mon[25696]: purged_snaps scrub starts 2024-04-14T15:22:37.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:37 smithi196 ceph-mon[25696]: purged_snaps scrub ok 2024-04-14T15:22:37.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:37 smithi196 ceph-mon[25696]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:37.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:37 smithi196 ceph-mon[25696]: osd.4 [v2:172.21.15.196:6800/61582609,v1:172.21.15.196:6801/61582609] boot 2024-04-14T15:22:37.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:37 smithi196 ceph-mon[25696]: osdmap e31: 5 total, 5 up, 5 in 2024-04-14T15:22:37.862 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:37 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:37.996 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-14T15:22:37.996 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-14T15:22:37.996 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-04-14T15:22:37.996 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-04-14T15:22:37.996 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126906 s, 826 MB/s 2024-04-14T15:22:37.996 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-04-14T15:22:38.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[29517]: purged_snaps scrub starts 2024-04-14T15:22:38.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[29517]: purged_snaps scrub ok 2024-04-14T15:22:38.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[29517]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:38.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[29517]: osd.4 [v2:172.21.15.196:6800/61582609,v1:172.21.15.196:6801/61582609] boot 2024-04-14T15:22:38.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[29517]: osdmap e31: 5 total, 5 up, 5 in 2024-04-14T15:22:38.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[24869]: purged_snaps scrub starts 2024-04-14T15:22:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[24869]: purged_snaps scrub ok 2024-04-14T15:22:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[24869]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T15:22:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[24869]: osd.4 [v2:172.21.15.196:6800/61582609,v1:172.21.15.196:6801/61582609] boot 2024-04-14T15:22:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[24869]: osdmap e31: 5 total, 5 up, 5 in 2024-04-14T15:22:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:37 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:22:38.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_3 2024-04-14T15:22:39.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:38 smithi016 ceph-mon[24869]: osdmap e32: 5 total, 5 up, 5 in 2024-04-14T15:22:39.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:38 smithi016 ceph-mon[29517]: osdmap e32: 5 total, 5 up, 5 in 2024-04-14T15:22:39.155 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:38 smithi196 ceph-mon[25696]: osdmap e32: 5 total, 5 up, 5 in 2024-04-14T15:22:39.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:39 smithi196 ceph-mon[25696]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:39.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:39 smithi196 ceph-mon[25696]: osdmap e33: 5 total, 5 up, 5 in 2024-04-14T15:22:39.942 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:39 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:39.943 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:39 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:39.943 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:39 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:39.943 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:39 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[24869]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[24869]: osdmap e33: 5 total, 5 up, 5 in 2024-04-14T15:22:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:40.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[29517]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:40.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[29517]: osdmap e33: 5 total, 5 up, 5 in 2024-04-14T15:22:40.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:40.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:22:40.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:40.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:39 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:40.822 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:40 smithi196 ceph-mon[25696]: Detected new or changed devices on smithi196 2024-04-14T15:22:40.822 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:40 smithi196 ceph-mon[25696]: Adjusting osd_memory_target on smithi196 to 17088M 2024-04-14T15:22:40.822 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:40 smithi196 ceph-mon[25696]: osdmap e34: 5 total, 5 up, 5 in 2024-04-14T15:22:41.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:40 smithi016 ceph-mon[24869]: Detected new or changed devices on smithi196 2024-04-14T15:22:41.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:40 smithi016 ceph-mon[24869]: Adjusting osd_memory_target on smithi196 to 17088M 2024-04-14T15:22:41.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:40 smithi016 ceph-mon[24869]: osdmap e34: 5 total, 5 up, 5 in 2024-04-14T15:22:41.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:40 smithi016 ceph-mon[29517]: Detected new or changed devices on smithi196 2024-04-14T15:22:41.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:40 smithi016 ceph-mon[29517]: Adjusting osd_memory_target on smithi196 to 17088M 2024-04-14T15:22:41.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:40 smithi016 ceph-mon[29517]: osdmap e34: 5 total, 5 up, 5 in 2024-04-14T15:22:41.885 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:41 smithi196 ceph-mon[25696]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-14T15:22:41.885 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:41 smithi196 ceph-mon[25696]: osdmap e35: 5 total, 5 up, 5 in 2024-04-14T15:22:41.885 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:41 smithi196 ceph-mon[25696]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:22:41.885 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:41 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:22:41.885 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:41 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:22:41.885 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:41 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:42.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[24869]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-14T15:22:42.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[24869]: osdmap e35: 5 total, 5 up, 5 in 2024-04-14T15:22:42.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[24869]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:22:42.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:22:42.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:22:42.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:42.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[29517]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-14T15:22:42.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[29517]: osdmap e35: 5 total, 5 up, 5 in 2024-04-14T15:22:42.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[29517]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:22:42.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:22:42.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:22:42.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:41 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:42.849 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:42 smithi196 ceph-mon[25696]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-14T15:22:43.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:42 smithi016 ceph-mon[24869]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-14T15:22:43.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:42 smithi016 ceph-mon[29517]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-14T15:22:44.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:43 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/4264310370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b"}]: dispatch 2024-04-14T15:22:44.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:43 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b"}]: dispatch 2024-04-14T15:22:44.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:43 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b"}]': finished 2024-04-14T15:22:44.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:43 smithi016 ceph-mon[24869]: osdmap e36: 6 total, 5 up, 6 in 2024-04-14T15:22:44.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:43 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:22:44.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:43 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/4264310370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b"}]: dispatch 2024-04-14T15:22:44.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:43 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b"}]: dispatch 2024-04-14T15:22:44.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:43 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b"}]': finished 2024-04-14T15:22:44.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:43 smithi016 ceph-mon[29517]: osdmap e36: 6 total, 5 up, 6 in 2024-04-14T15:22:44.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:43 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:22:44.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:43 smithi196 ceph-mon[25696]: from='client.? 172.21.15.196:0/4264310370' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b"}]: dispatch 2024-04-14T15:22:44.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:43 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b"}]: dispatch 2024-04-14T15:22:44.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:43 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b"}]': finished 2024-04-14T15:22:44.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:43 smithi196 ceph-mon[25696]: osdmap e36: 6 total, 5 up, 6 in 2024-04-14T15:22:44.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:43 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:22:45.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:44 smithi016 ceph-mon[24869]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-14T15:22:45.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:44 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/2542443845' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:22:45.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:44 smithi016 ceph-mon[29517]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-14T15:22:45.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:44 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/2542443845' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:22:45.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:44 smithi196 ceph-mon[25696]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-14T15:22:45.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:44 smithi196 ceph-mon[25696]: from='client.? 172.21.15.196:0/2542443845' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:22:46.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:45 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:22:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:45 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:22:46.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:22:45 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:22:45.712+0000 7fbcd6501700 -1 calc_pg_upmaps abort due to max <= 0 2024-04-14T15:22:46.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:45 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:22:46.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:45 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:22:46.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:45 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:22:46.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:45 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:22:46.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:46 smithi196 ceph-mon[25696]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:47.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:46 smithi016 ceph-mon[24869]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:47.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:46 smithi016 ceph-mon[29517]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:49.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:49 smithi196 ceph-mon[25696]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:49 smithi016 ceph-mon[24869]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:49 smithi016 ceph-mon[29517]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:50.940 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:50 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:22:50.940 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:50 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:50.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:50 smithi196 ceph-mon[25696]: Deploying daemon osd.5 on smithi196 2024-04-14T15:22:51.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:50 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:22:51.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:50 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:51.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:50 smithi016 ceph-mon[29517]: Deploying daemon osd.5 on smithi196 2024-04-14T15:22:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:50 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:22:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:50 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:50 smithi016 ceph-mon[24869]: Deploying daemon osd.5 on smithi196 2024-04-14T15:22:51.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:51 smithi196 ceph-mon[25696]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:52.011 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:51 smithi016 ceph-mon[24869]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:52.012 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:51 smithi016 ceph-mon[29517]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:53.963 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:53 smithi196 ceph-mon[25696]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:54.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:53 smithi016 ceph-mon[29517]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:54.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:53 smithi016 ceph-mon[24869]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:55.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:55 smithi016 ceph-mon[29517]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:55.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:55 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:55.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:55 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:55.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:55 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:55.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:55 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:55.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:55 smithi016 ceph-mon[24869]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:55.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:55 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:55.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:55 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:55.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:55 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:55.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:55 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:55 smithi196 ceph-mon[25696]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:55 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:55 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:55 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:55 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:57.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:57 smithi016 ceph-mon[24869]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:57.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:57 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:57.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:57 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:57.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:57 smithi016 ceph-mon[29517]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:57.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:57 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:57.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:57 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:57.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:57 smithi196 ceph-mon[25696]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:57.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:57 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:57.868 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:57 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:57.898 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 5 on host 'smithi196' 2024-04-14T15:22:58.521 DEBUG:teuthology.orchestra.run.smithi196:osd.5> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.5.service 2024-04-14T15:22:58.523 INFO:tasks.cephadm:Deploying osd.6 on smithi196 with /dev/vg_nvme/lv_2... 2024-04-14T15:22:58.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-14T15:22:58.747 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:58 smithi196 ceph-mon[25696]: from='osd.5 [v2:172.21.15.196:6808/786436243,v1:172.21.15.196:6809/786436243]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:22:58.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:58 smithi196 ceph-mon[25696]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:22:58.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:58 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:58.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:58 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:58.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:58 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:58.748 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:58 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:58.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[24869]: from='osd.5 [v2:172.21.15.196:6808/786436243,v1:172.21.15.196:6809/786436243]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:22:58.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[24869]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:22:58.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:58.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:58.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:58.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:58.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[29517]: from='osd.5 [v2:172.21.15.196:6808/786436243,v1:172.21.15.196:6809/786436243]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:22:58.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[29517]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:22:58.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:22:58.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:22:58.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:22:58.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:58 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:22:59.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:59 smithi196 ceph-mon[25696]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:59.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:59 smithi196 ceph-mon[25696]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T15:22:59.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:59 smithi196 ceph-mon[25696]: osdmap e37: 6 total, 5 up, 6 in 2024-04-14T15:22:59.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:22:59.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:59 smithi196 ceph-mon[25696]: from='osd.5 [v2:172.21.15.196:6808/786436243,v1:172.21.15.196:6809/786436243]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:22:59.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:22:59 smithi196 ceph-mon[25696]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:22:59.779 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:22:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[32019]: 2024-04-14T15:22:59.581+0000 7f755c972700 -1 osd.5 0 waiting for initial osdmap 2024-04-14T15:22:59.780 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:22:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[32019]: 2024-04-14T15:22:59.586+0000 7f7556307700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:22:59.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[24869]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:59.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[24869]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T15:22:59.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[24869]: osdmap e37: 6 total, 5 up, 6 in 2024-04-14T15:22:59.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:22:59.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[24869]: from='osd.5 [v2:172.21.15.196:6808/786436243,v1:172.21.15.196:6809/786436243]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:22:59.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[24869]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:22:59.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[29517]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:22:59.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[29517]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T15:22:59.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[29517]: osdmap e37: 6 total, 5 up, 6 in 2024-04-14T15:22:59.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:22:59.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[29517]: from='osd.5 [v2:172.21.15.196:6808/786436243,v1:172.21.15.196:6809/786436243]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:22:59.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:22:59 smithi016 ceph-mon[29517]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:00.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:00 smithi196 ceph-mon[25696]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:23:00.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:00 smithi196 ceph-mon[25696]: osdmap e38: 6 total, 5 up, 6 in 2024-04-14T15:23:00.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:00 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:00.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:00 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:00 smithi016 ceph-mon[24869]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:23:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:00 smithi016 ceph-mon[24869]: osdmap e38: 6 total, 5 up, 6 in 2024-04-14T15:23:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:00 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:00 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:00.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:00 smithi016 ceph-mon[29517]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:23:00.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:00 smithi016 ceph-mon[29517]: osdmap e38: 6 total, 5 up, 6 in 2024-04-14T15:23:00.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:00 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:00.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:00 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:01.142 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-14T15:23:01.142 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-14T15:23:01.142 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-04-14T15:23:01.142 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-04-14T15:23:01.142 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125326 s, 837 MB/s 2024-04-14T15:23:01.142 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-04-14T15:23:01.806 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:01 smithi196 ceph-mon[25696]: purged_snaps scrub starts 2024-04-14T15:23:01.806 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:01 smithi196 ceph-mon[25696]: purged_snaps scrub ok 2024-04-14T15:23:01.806 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:01 smithi196 ceph-mon[25696]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:23:01.806 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:01 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:01.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:01 smithi196 ceph-mon[25696]: osd.5 [v2:172.21.15.196:6808/786436243,v1:172.21.15.196:6809/786436243] boot 2024-04-14T15:23:01.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:01 smithi196 ceph-mon[25696]: osdmap e39: 6 total, 6 up, 6 in 2024-04-14T15:23:01.807 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:01 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:01.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_2 2024-04-14T15:23:01.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[24869]: purged_snaps scrub starts 2024-04-14T15:23:01.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[24869]: purged_snaps scrub ok 2024-04-14T15:23:01.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[24869]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:23:01.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:01.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[24869]: osd.5 [v2:172.21.15.196:6808/786436243,v1:172.21.15.196:6809/786436243] boot 2024-04-14T15:23:01.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[24869]: osdmap e39: 6 total, 6 up, 6 in 2024-04-14T15:23:01.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[29517]: purged_snaps scrub starts 2024-04-14T15:23:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[29517]: purged_snaps scrub ok 2024-04-14T15:23:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[29517]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T15:23:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[29517]: osd.5 [v2:172.21.15.196:6808/786436243,v1:172.21.15.196:6809/786436243] boot 2024-04-14T15:23:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[29517]: osdmap e39: 6 total, 6 up, 6 in 2024-04-14T15:23:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:01 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:23:02.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:02 smithi196 ceph-mon[25696]: osdmap e40: 6 total, 6 up, 6 in 2024-04-14T15:23:02.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:02 smithi016 ceph-mon[24869]: osdmap e40: 6 total, 6 up, 6 in 2024-04-14T15:23:02.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:02 smithi016 ceph-mon[29517]: osdmap e40: 6 total, 6 up, 6 in 2024-04-14T15:23:03.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:03 smithi196 ceph-mon[25696]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:03.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:03 smithi196 ceph-mon[25696]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T15:23:03.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:03 smithi196 ceph-mon[25696]: osdmap e41: 6 total, 6 up, 6 in 2024-04-14T15:23:03.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:03 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:03.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:03 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:03.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:03 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:03.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:03 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:03.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:03 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:03.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[24869]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:03.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[24869]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[24869]: osdmap e41: 6 total, 6 up, 6 in 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[29517]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[29517]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[29517]: osdmap e41: 6 total, 6 up, 6 in 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:03.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:03 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:04.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[24869]: Detected new or changed devices on smithi196 2024-04-14T15:23:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[24869]: Adjusting osd_memory_target on smithi196 to 8544M 2024-04-14T15:23:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[24869]: osdmap e42: 6 total, 6 up, 6 in 2024-04-14T15:23:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:23:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:23:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[29517]: Detected new or changed devices on smithi196 2024-04-14T15:23:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[29517]: Adjusting osd_memory_target on smithi196 to 8544M 2024-04-14T15:23:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[29517]: osdmap e42: 6 total, 6 up, 6 in 2024-04-14T15:23:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:23:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:23:04.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:04 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:04.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:04 smithi196 ceph-mon[25696]: Detected new or changed devices on smithi196 2024-04-14T15:23:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:04 smithi196 ceph-mon[25696]: Adjusting osd_memory_target on smithi196 to 8544M 2024-04-14T15:23:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:04 smithi196 ceph-mon[25696]: osdmap e42: 6 total, 6 up, 6 in 2024-04-14T15:23:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:04 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:23:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:04 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:23:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:04 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:05.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:05 smithi016 ceph-mon[24869]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:05.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:05 smithi016 ceph-mon[24869]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:23:05.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:05 smithi016 ceph-mon[24869]: osdmap e43: 6 total, 6 up, 6 in 2024-04-14T15:23:05.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:05 smithi016 ceph-mon[29517]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:05.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:05 smithi016 ceph-mon[29517]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:23:05.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:05 smithi016 ceph-mon[29517]: osdmap e43: 6 total, 6 up, 6 in 2024-04-14T15:23:05.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:05 smithi196 ceph-mon[25696]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:05.863 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:05 smithi196 ceph-mon[25696]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:23:05.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:05 smithi196 ceph-mon[25696]: osdmap e43: 6 total, 6 up, 6 in 2024-04-14T15:23:07.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[24869]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/2006352148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175090a7-f054-4e4d-8c65-d9b503e6c547"}]: dispatch 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175090a7-f054-4e4d-8c65-d9b503e6c547"}]: dispatch 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "175090a7-f054-4e4d-8c65-d9b503e6c547"}]': finished 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[24869]: osdmap e44: 7 total, 6 up, 7 in 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/1231865488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[29517]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/2006352148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175090a7-f054-4e4d-8c65-d9b503e6c547"}]: dispatch 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175090a7-f054-4e4d-8c65-d9b503e6c547"}]: dispatch 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "175090a7-f054-4e4d-8c65-d9b503e6c547"}]': finished 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[29517]: osdmap e44: 7 total, 6 up, 7 in 2024-04-14T15:23:07.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:07.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:07 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/1231865488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:23:07.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:07 smithi196 ceph-mon[25696]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:07.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:07 smithi196 ceph-mon[25696]: from='client.? 172.21.15.196:0/2006352148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175090a7-f054-4e4d-8c65-d9b503e6c547"}]: dispatch 2024-04-14T15:23:07.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:07 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175090a7-f054-4e4d-8c65-d9b503e6c547"}]: dispatch 2024-04-14T15:23:07.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:07 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "175090a7-f054-4e4d-8c65-d9b503e6c547"}]': finished 2024-04-14T15:23:07.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:07 smithi196 ceph-mon[25696]: osdmap e44: 7 total, 6 up, 7 in 2024-04-14T15:23:07.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:07 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:07.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:07 smithi196 ceph-mon[25696]: from='client.? 172.21.15.196:0/1231865488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:23:08.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:08 smithi196 ceph-mon[25696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T15:23:08.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:08 smithi196 ceph-mon[25696]: Cluster is now healthy 2024-04-14T15:23:09.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:08 smithi016 ceph-mon[24869]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T15:23:09.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:08 smithi016 ceph-mon[24869]: Cluster is now healthy 2024-04-14T15:23:09.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:08 smithi016 ceph-mon[29517]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T15:23:09.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:08 smithi016 ceph-mon[29517]: Cluster is now healthy 2024-04-14T15:23:09.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:09 smithi196 ceph-mon[25696]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-14T15:23:10.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:09 smithi016 ceph-mon[24869]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-14T15:23:10.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:09 smithi016 ceph-mon[29517]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-14T15:23:11.827 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:11 smithi196 ceph-mon[25696]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-14T15:23:12.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:11 smithi016 ceph-mon[24869]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-14T15:23:12.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:11 smithi016 ceph-mon[29517]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-14T15:23:13.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:13 smithi196 ceph-mon[25696]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T15:23:13.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:13 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:23:13.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:13 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:14.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:13 smithi016 ceph-mon[24869]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T15:23:14.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:13 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:23:14.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:13 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:14.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:13 smithi016 ceph-mon[29517]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T15:23:14.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:13 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:23:14.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:13 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:14.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:14 smithi196 ceph-mon[25696]: Deploying daemon osd.6 on smithi196 2024-04-14T15:23:15.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:14 smithi016 ceph-mon[24869]: Deploying daemon osd.6 on smithi196 2024-04-14T15:23:15.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:14 smithi016 ceph-mon[29517]: Deploying daemon osd.6 on smithi196 2024-04-14T15:23:16.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:15 smithi196 ceph-mon[25696]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-14T15:23:16.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:15 smithi016 ceph-mon[24869]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-14T15:23:16.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:15 smithi016 ceph-mon[29517]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-14T15:23:17.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:16 smithi196 ceph-mon[25696]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T15:23:17.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:16 smithi016 ceph-mon[24869]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T15:23:17.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:16 smithi016 ceph-mon[29517]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T15:23:19.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:19 smithi196 ceph-mon[25696]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:19.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:19 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:19.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:19 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:19.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:19 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:19.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:19 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:19.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:19 smithi016 ceph-mon[29517]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:19.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:19 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:19.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:19 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:19.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:19 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:19.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:19 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:19.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:19 smithi016 ceph-mon[24869]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:19.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:19 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:19.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:19 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:19.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:19 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:19.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:19 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:21.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:21 smithi196 ceph-mon[25696]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:21.355 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 6 on host 'smithi196' 2024-04-14T15:23:21.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:21 smithi016 ceph-mon[24869]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:21.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:21 smithi016 ceph-mon[29517]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:22.069 DEBUG:teuthology.orchestra.run.smithi196:osd.6> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.6.service 2024-04-14T15:23:22.071 INFO:tasks.cephadm:Deploying osd.7 on smithi196 with /dev/vg_nvme/lv_1... 2024-04-14T15:23:22.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-14T15:23:22.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:22 smithi196 ceph-mon[25696]: from='osd.6 [v2:172.21.15.196:6816/3846591221,v1:172.21.15.196:6817/3846591221]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:23:22.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:22 smithi196 ceph-mon[25696]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:23:22.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:22 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:22.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:22 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:22.512 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:22 smithi016 ceph-mon[29517]: from='osd.6 [v2:172.21.15.196:6816/3846591221,v1:172.21.15.196:6817/3846591221]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:23:22.512 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:22 smithi016 ceph-mon[29517]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:23:22.512 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:22 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:22.512 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:22 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:22.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:22 smithi016 ceph-mon[24869]: from='osd.6 [v2:172.21.15.196:6816/3846591221,v1:172.21.15.196:6817/3846591221]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:23:22.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:22 smithi016 ceph-mon[24869]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:23:22.512 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:22 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:22.513 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:22 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:23.458 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:23 smithi196 ceph-mon[25696]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:23.458 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:23 smithi196 ceph-mon[25696]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T15:23:23.458 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:23 smithi196 ceph-mon[25696]: osdmap e45: 7 total, 6 up, 7 in 2024-04-14T15:23:23.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:23 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:23.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:23 smithi196 ceph-mon[25696]: from='osd.6 [v2:172.21.15.196:6816/3846591221,v1:172.21.15.196:6817/3846591221]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:23.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:23 smithi196 ceph-mon[25696]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:23.459 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:23:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[34914]: 2024-04-14T15:23:23.194+0000 7f8061d36700 -1 osd.6 0 waiting for initial osdmap 2024-04-14T15:23:23.459 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:23:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[34914]: 2024-04-14T15:23:23.202+0000 7f805becc700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:23:23.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[24869]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:23.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[24869]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T15:23:23.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[24869]: osdmap e45: 7 total, 6 up, 7 in 2024-04-14T15:23:23.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:23.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[24869]: from='osd.6 [v2:172.21.15.196:6816/3846591221,v1:172.21.15.196:6817/3846591221]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:23.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[24869]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:23.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[29517]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:23.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[29517]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T15:23:23.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[29517]: osdmap e45: 7 total, 6 up, 7 in 2024-04-14T15:23:23.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:23.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[29517]: from='osd.6 [v2:172.21.15.196:6816/3846591221,v1:172.21.15.196:6817/3846591221]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:23.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:23 smithi016 ceph-mon[29517]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-14T15:23:24.069 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0141898 s, 739 MB/s 2024-04-14T15:23:24.070 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-04-14T15:23:24.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:24 smithi196 ceph-mon[25696]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:23:24.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:24 smithi196 ceph-mon[25696]: osdmap e46: 7 total, 6 up, 7 in 2024-04-14T15:23:24.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:24 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:24.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:24 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:24.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:24 smithi016 ceph-mon[24869]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:23:24.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:24 smithi016 ceph-mon[24869]: osdmap e46: 7 total, 6 up, 7 in 2024-04-14T15:23:24.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:24 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:24.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:24 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:24.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:24 smithi016 ceph-mon[29517]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:23:24.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:24 smithi016 ceph-mon[29517]: osdmap e46: 7 total, 6 up, 7 in 2024-04-14T15:23:24.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:24 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:24.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:24 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:25.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:25 smithi196 ceph-mon[25696]: purged_snaps scrub starts 2024-04-14T15:23:25.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:25 smithi196 ceph-mon[25696]: purged_snaps scrub ok 2024-04-14T15:23:25.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:25 smithi196 ceph-mon[25696]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:25.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:25 smithi196 ceph-mon[25696]: osd.6 [v2:172.21.15.196:6816/3846591221,v1:172.21.15.196:6817/3846591221] boot 2024-04-14T15:23:25.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:25 smithi196 ceph-mon[25696]: osdmap e47: 7 total, 7 up, 7 in 2024-04-14T15:23:25.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:25 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:25.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi196:vg_nvme/lv_1 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[24869]: purged_snaps scrub starts 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[24869]: purged_snaps scrub ok 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[24869]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[24869]: osd.6 [v2:172.21.15.196:6816/3846591221,v1:172.21.15.196:6817/3846591221] boot 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[24869]: osdmap e47: 7 total, 7 up, 7 in 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[29517]: purged_snaps scrub starts 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[29517]: purged_snaps scrub ok 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[29517]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[29517]: osd.6 [v2:172.21.15.196:6816/3846591221,v1:172.21.15.196:6817/3846591221] boot 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[29517]: osdmap e47: 7 total, 7 up, 7 in 2024-04-14T15:23:25.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:25 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:23:26.345 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:26 smithi196 ceph-mon[25696]: osdmap e48: 7 total, 7 up, 7 in 2024-04-14T15:23:26.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:26 smithi016 ceph-mon[24869]: osdmap e48: 7 total, 7 up, 7 in 2024-04-14T15:23:26.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:26 smithi016 ceph-mon[29517]: osdmap e48: 7 total, 7 up, 7 in 2024-04-14T15:23:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: osdmap e49: 7 total, 7 up, 7 in 2024-04-14T15:23:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:27.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:27.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:27.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:27 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: osdmap e49: 7 total, 7 up, 7 in 2024-04-14T15:23:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: osdmap e49: 7 total, 7 up, 7 in 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:27 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:28.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:28 smithi196 ceph-mon[25696]: Detected new or changed devices on smithi196 2024-04-14T15:23:28.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:28 smithi196 ceph-mon[25696]: Adjusting osd_memory_target on smithi196 to 5696M 2024-04-14T15:23:28.538 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:28 smithi196 ceph-mon[25696]: osdmap e50: 7 total, 7 up, 7 in 2024-04-14T15:23:28.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:28 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:23:28.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:28 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:23:28.539 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:28 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:28.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[24869]: Detected new or changed devices on smithi196 2024-04-14T15:23:28.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[24869]: Adjusting osd_memory_target on smithi196 to 5696M 2024-04-14T15:23:28.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[24869]: osdmap e50: 7 total, 7 up, 7 in 2024-04-14T15:23:28.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:23:28.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:23:28.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[29517]: Detected new or changed devices on smithi196 2024-04-14T15:23:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[29517]: Adjusting osd_memory_target on smithi196 to 5696M 2024-04-14T15:23:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[29517]: osdmap e50: 7 total, 7 up, 7 in 2024-04-14T15:23:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T15:23:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T15:23:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:28 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:29.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:29 smithi196 ceph-mon[25696]: pgmap v151: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-14T15:23:29.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:29 smithi196 ceph-mon[25696]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:23:29.499 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:29 smithi196 ceph-mon[25696]: osdmap e51: 7 total, 7 up, 7 in 2024-04-14T15:23:29.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:29 smithi016 ceph-mon[24869]: pgmap v151: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-14T15:23:29.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:29 smithi016 ceph-mon[24869]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:23:29.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:29 smithi016 ceph-mon[24869]: osdmap e51: 7 total, 7 up, 7 in 2024-04-14T15:23:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:29 smithi016 ceph-mon[29517]: pgmap v151: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-14T15:23:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:29 smithi016 ceph-mon[29517]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:23:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:29 smithi016 ceph-mon[29517]: osdmap e51: 7 total, 7 up, 7 in 2024-04-14T15:23:31.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[24869]: pgmap v153: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-14T15:23:31.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/1471803134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2161a137-f1f6-473c-93d2-39ede9f5d6a5"}]: dispatch 2024-04-14T15:23:31.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2161a137-f1f6-473c-93d2-39ede9f5d6a5"}]: dispatch 2024-04-14T15:23:31.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[24869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2161a137-f1f6-473c-93d2-39ede9f5d6a5"}]': finished 2024-04-14T15:23:31.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[24869]: osdmap e52: 8 total, 7 up, 8 in 2024-04-14T15:23:31.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:31.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/3082107760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:23:31.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[29517]: pgmap v153: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-14T15:23:31.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/1471803134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2161a137-f1f6-473c-93d2-39ede9f5d6a5"}]: dispatch 2024-04-14T15:23:31.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2161a137-f1f6-473c-93d2-39ede9f5d6a5"}]: dispatch 2024-04-14T15:23:31.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[29517]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2161a137-f1f6-473c-93d2-39ede9f5d6a5"}]': finished 2024-04-14T15:23:31.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[29517]: osdmap e52: 8 total, 7 up, 8 in 2024-04-14T15:23:31.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:31.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:31 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/3082107760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:23:31.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:31 smithi196 ceph-mon[25696]: pgmap v153: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-14T15:23:31.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:31 smithi196 ceph-mon[25696]: from='client.? 172.21.15.196:0/1471803134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2161a137-f1f6-473c-93d2-39ede9f5d6a5"}]: dispatch 2024-04-14T15:23:31.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:31 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2161a137-f1f6-473c-93d2-39ede9f5d6a5"}]: dispatch 2024-04-14T15:23:31.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:31 smithi196 ceph-mon[25696]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2161a137-f1f6-473c-93d2-39ede9f5d6a5"}]': finished 2024-04-14T15:23:31.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:31 smithi196 ceph-mon[25696]: osdmap e52: 8 total, 7 up, 8 in 2024-04-14T15:23:31.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:31 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:31.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:31 smithi196 ceph-mon[25696]: from='client.? 172.21.15.196:0/3082107760' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T15:23:33.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:33 smithi016 ceph-mon[24869]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-14T15:23:33.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:33 smithi016 ceph-mon[29517]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-14T15:23:33.629 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:33 smithi196 ceph-mon[25696]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-14T15:23:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:35 smithi196 ceph-mon[25696]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:36.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:35 smithi016 ceph-mon[24869]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:35 smithi016 ceph-mon[29517]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:37 smithi196 ceph-mon[25696]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:37 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:23:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:37 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:38.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:37 smithi016 ceph-mon[24869]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:38.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:37 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:23:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:37 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:38.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:37 smithi016 ceph-mon[29517]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:38.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:37 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:23:38.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:37 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:38.846 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:38 smithi196 ceph-mon[25696]: Deploying daemon osd.7 on smithi196 2024-04-14T15:23:39.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:38 smithi016 ceph-mon[24869]: Deploying daemon osd.7 on smithi196 2024-04-14T15:23:39.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:38 smithi016 ceph-mon[29517]: Deploying daemon osd.7 on smithi196 2024-04-14T15:23:39.989 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:39 smithi196 ceph-mon[25696]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:40.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:39 smithi016 ceph-mon[24869]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:40.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:39 smithi016 ceph-mon[29517]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:41.848 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:41 smithi196 ceph-mon[25696]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:42.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:41 smithi016 ceph-mon[24869]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:42.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:41 smithi016 ceph-mon[29517]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:43.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:42 smithi196 ceph-mon[25696]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:43.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:42 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:43.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:42 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:43.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:42 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:43.088 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:42 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:43.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:42 smithi016 ceph-mon[24869]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:43.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:42 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:43.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:42 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:43.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:42 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:43.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:42 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:43.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:42 smithi016 ceph-mon[29517]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:43.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:42 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:43.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:42 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:43.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:42 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:43.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:42 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:44.838 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 7 on host 'smithi196' 2024-04-14T15:23:45.492 DEBUG:teuthology.orchestra.run.smithi196:osd.7> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.7.service 2024-04-14T15:23:45.495 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-14T15:23:45.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd stat -f json 2024-04-14T15:23:45.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:45 smithi196 ceph-mon[25696]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:45.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:45 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:45.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:45 smithi196 ceph-mon[25696]: from='osd.7 [v2:172.21.15.196:6824/1305411055,v1:172.21.15.196:6825/1305411055]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:23:45.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:45 smithi196 ceph-mon[25696]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:23:45.737 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:45 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:45.818 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:45 smithi016 ceph-mon[29517]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:45.818 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:45 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:45.818 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:45 smithi016 ceph-mon[29517]: from='osd.7 [v2:172.21.15.196:6824/1305411055,v1:172.21.15.196:6825/1305411055]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:23:45.818 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:45 smithi016 ceph-mon[29517]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:23:45.818 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:45 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:45.818 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:45 smithi016 ceph-mon[24869]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:45.818 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:45 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:45.818 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:45 smithi016 ceph-mon[24869]: from='osd.7 [v2:172.21.15.196:6824/1305411055,v1:172.21.15.196:6825/1305411055]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:23:45.818 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:45 smithi016 ceph-mon[24869]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:23:45.818 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:45 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:47.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:23:47.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:23:47.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:46 smithi196 ceph-mon[25696]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T15:23:47.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:46 smithi196 ceph-mon[25696]: osdmap e53: 8 total, 7 up, 8 in 2024-04-14T15:23:47.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:46 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:47.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:46 smithi196 ceph-mon[25696]: from='osd.7 [v2:172.21.15.196:6824/1305411055,v1:172.21.15.196:6825/1305411055]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:47.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:46 smithi196 ceph-mon[25696]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:47.051 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:23:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[37913]: 2024-04-14T15:23:46.844+0000 7fdf5bd91700 -1 osd.7 0 waiting for initial osdmap 2024-04-14T15:23:47.051 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:23:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[37913]: 2024-04-14T15:23:46.853+0000 7fdf55f27700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:23:47.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:23:47.062 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:23:47.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[24869]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T15:23:47.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[24869]: osdmap e53: 8 total, 7 up, 8 in 2024-04-14T15:23:47.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:47.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[24869]: from='osd.7 [v2:172.21.15.196:6824/1305411055,v1:172.21.15.196:6825/1305411055]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:47.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[24869]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:47.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:23:47.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:23:47.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[29517]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T15:23:47.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[29517]: osdmap e53: 8 total, 7 up, 8 in 2024-04-14T15:23:47.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:47.064 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[29517]: from='osd.7 [v2:172.21.15.196:6824/1305411055,v1:172.21.15.196:6825/1305411055]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:47.064 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:46 smithi016 ceph-mon[29517]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:23:47.571 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:23:48.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:47 smithi196 ceph-mon[25696]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:48.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:47 smithi196 ceph-mon[25696]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:23:48.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:47 smithi196 ceph-mon[25696]: osdmap e54: 8 total, 7 up, 8 in 2024-04-14T15:23:48.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:47 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:48.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:47 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:48.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:47 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/901097213' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T15:23:48.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[29517]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:48.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[29517]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:23:48.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[29517]: osdmap e54: 8 total, 7 up, 8 in 2024-04-14T15:23:48.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:48.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:48.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/901097213' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T15:23:48.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[24869]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:48.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[24869]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-04-14T15:23:48.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[24869]: osdmap e54: 8 total, 7 up, 8 in 2024-04-14T15:23:48.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:48.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:48.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:47 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/901097213' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T15:23:48.185 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":54,"num_osds":8,"num_up_osds":7,"osd_up_since":1713108204,"num_in_osds":8,"osd_in_since":1713108210,"num_remapped_pgs":0} 2024-04-14T15:23:49.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[29517]: purged_snaps scrub starts 2024-04-14T15:23:49.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[29517]: purged_snaps scrub ok 2024-04-14T15:23:49.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[29517]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:49.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[29517]: osd.7 [v2:172.21.15.196:6824/1305411055,v1:172.21.15.196:6825/1305411055] boot 2024-04-14T15:23:49.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[29517]: osdmap e55: 8 total, 8 up, 8 in 2024-04-14T15:23:49.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:49.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[29517]: osdmap e56: 8 total, 8 up, 8 in 2024-04-14T15:23:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[24869]: purged_snaps scrub starts 2024-04-14T15:23:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[24869]: purged_snaps scrub ok 2024-04-14T15:23:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[24869]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[24869]: osd.7 [v2:172.21.15.196:6824/1305411055,v1:172.21.15.196:6825/1305411055] boot 2024-04-14T15:23:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[24869]: osdmap e55: 8 total, 8 up, 8 in 2024-04-14T15:23:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:48 smithi016 ceph-mon[24869]: osdmap e56: 8 total, 8 up, 8 in 2024-04-14T15:23:49.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:48 smithi196 ceph-mon[25696]: purged_snaps scrub starts 2024-04-14T15:23:49.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:48 smithi196 ceph-mon[25696]: purged_snaps scrub ok 2024-04-14T15:23:49.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:48 smithi196 ceph-mon[25696]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T15:23:49.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:48 smithi196 ceph-mon[25696]: osd.7 [v2:172.21.15.196:6824/1305411055,v1:172.21.15.196:6825/1305411055] boot 2024-04-14T15:23:49.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:48 smithi196 ceph-mon[25696]: osdmap e55: 8 total, 8 up, 8 in 2024-04-14T15:23:49.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:48 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:23:49.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:48 smithi196 ceph-mon[25696]: osdmap e56: 8 total, 8 up, 8 in 2024-04-14T15:23:49.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd stat -f json 2024-04-14T15:23:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: Detected new or changed devices on smithi196 2024-04-14T15:23:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: Adjusting osd_memory_target on smithi196 to 4272M 2024-04-14T15:23:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[24869]: osdmap e57: 8 total, 8 up, 8 in 2024-04-14T15:23:50.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: Detected new or changed devices on smithi196 2024-04-14T15:23:50.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: Adjusting osd_memory_target on smithi196 to 4272M 2024-04-14T15:23:50.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:50.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:50.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:50.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:49 smithi016 ceph-mon[29517]: osdmap e57: 8 total, 8 up, 8 in 2024-04-14T15:23:50.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: Detected new or changed devices on smithi196 2024-04-14T15:23:50.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:23:50.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: Adjusting osd_memory_target on smithi196 to 4272M 2024-04-14T15:23:50.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:23:50.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:23:50.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:23:50.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:23:50.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:49 smithi196 ceph-mon[25696]: osdmap e57: 8 total, 8 up, 8 in 2024-04-14T15:23:51.103 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:23:51.773 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:51 smithi016 ceph-mon[24869]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:23:51.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:51 smithi016 ceph-mon[24869]: osdmap e58: 8 total, 8 up, 8 in 2024-04-14T15:23:51.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:51 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1773326866' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T15:23:51.774 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:51 smithi016 ceph-mon[29517]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:23:51.774 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:51 smithi016 ceph-mon[29517]: osdmap e58: 8 total, 8 up, 8 in 2024-04-14T15:23:51.774 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:51 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1773326866' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T15:23:51.799 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1713108227,"num_in_osds":8,"osd_in_since":1713108210,"num_remapped_pgs":0} 2024-04-14T15:23:51.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T15:23:52.021 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:52.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:51 smithi196 ceph-mon[25696]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:23:52.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:51 smithi196 ceph-mon[25696]: osdmap e58: 8 total, 8 up, 8 in 2024-04-14T15:23:52.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:51 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1773326866' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T15:23:53.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:52 smithi016 ceph-mon[24869]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:23:53.089 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:52 smithi016 ceph-mon[24869]: osdmap e59: 8 total, 8 up, 8 in 2024-04-14T15:23:53.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:52 smithi016 ceph-mon[29517]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:23:53.089 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:52 smithi016 ceph-mon[29517]: osdmap e59: 8 total, 8 up, 8 in 2024-04-14T15:23:53.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:52 smithi196 ceph-mon[25696]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:23:53.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:52 smithi196 ceph-mon[25696]: osdmap e59: 8 total, 8 up, 8 in 2024-04-14T15:23:53.738 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:23:53.738 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":59,"fsid":"3ca5917a-fa72-11ee-bc8e-c7b262605968","created":"2024-04-14T15:18:59.224477+0000","modified":"2024-04-14T15:23:51.910407+0000","last_up_change":"2024-04-14T15:23:47.838098+0000","last_in_change":"2024-04-14T15:23:30.452634+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T15:21:54.002288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b8767d4c-2155-4a83-8522-ac3042eaa0be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6803","nonce":696303661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6805","nonce":696303661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6809","nonce":696303661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6807","nonce":696303661}]},"public_addr":"172.21.15.16:6803/696303661","cluster_addr":"172.21.15.16:6805/696303661","heartbeat_back_addr":"172.21.15.16:6809/696303661","heartbeat_front_addr":"172.21.15.16:6807/696303661","state":["exists","up"]},{"osd":1,"uuid":"ef5e2c3d-9d71-41c3-a640-188d71c4fe72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6811","nonce":1936311739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6813","nonce":1936311739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6817","nonce":1936311739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6815","nonce":1936311739}]},"public_addr":"172.21.15.16:6811/1936311739","cluster_addr":"172.21.15.16:6813/1936311739","heartbeat_back_addr":"172.21.15.16:6817/1936311739","heartbeat_front_addr":"172.21.15.16:6815/1936311739","state":["exists","up"]},{"osd":2,"uuid":"0b1e7e85-29f9-4b4f-a16c-d80ed36951c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6819","nonce":701104650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6821","nonce":701104650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6825","nonce":701104650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6823","nonce":701104650}]},"public_addr":"172.21.15.16:6819/701104650","cluster_addr":"172.21.15.16:6821/701104650","heartbeat_back_addr":"172.21.15.16:6825/701104650","heartbeat_front_addr":"172.21.15.16:6823/701104650","state":["exists","up"]},{"osd":3,"uuid":"5f943ae4-952d-4e12-97c5-2a23c47a1c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6827","nonce":3777247940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6829","nonce":3777247940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6833","nonce":3777247940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6831","nonce":3777247940}]},"public_addr":"172.21.15.16:6827/3777247940","cluster_addr":"172.21.15.16:6829/3777247940","heartbeat_back_addr":"172.21.15.16:6833/3777247940","heartbeat_front_addr":"172.21.15.16:6831/3777247940","state":["exists","up"]},{"osd":4,"uuid":"1ad25b86-55ee-4769-991e-de2a723b3516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6801","nonce":61582609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6803","nonce":61582609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6807","nonce":61582609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6805","nonce":61582609}]},"public_addr":"172.21.15.196:6801/61582609","cluster_addr":"172.21.15.196:6803/61582609","heartbeat_back_addr":"172.21.15.196:6807/61582609","heartbeat_front_addr":"172.21.15.196:6805/61582609","state":["exists","up"]},{"osd":5,"uuid":"185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6809","nonce":786436243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6811","nonce":786436243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6815","nonce":786436243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6813","nonce":786436243}]},"public_addr":"172.21.15.196:6809/786436243","cluster_addr":"172.21.15.196:6811/786436243","heartbeat_back_addr":"172.21.15.196:6815/786436243","heartbeat_front_addr":"172.21.15.196:6813/786436243","state":["exists","up"]},{"osd":6,"uuid":"175090a7-f054-4e4d-8c65-d9b503e6c547","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6817","nonce":3846591221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6819","nonce":3846591221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6823","nonce":3846591221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6821","nonce":3846591221}]},"public_addr":"172.21.15.196:6817/3846591221","cluster_addr":"172.21.15.196:6819/3846591221","heartbeat_back_addr":"172.21.15.196:6823/3846591221","heartbeat_front_addr":"172.21.15.196:6821/3846591221","state":["exists","up"]},{"osd":7,"uuid":"2161a137-f1f6-473c-93d2-39ede9f5d6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6825","nonce":1305411055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6827","nonce":1305411055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6831","nonce":1305411055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6829","nonce":1305411055}]},"public_addr":"172.21.15.196:6825/1305411055","cluster_addr":"172.21.15.196:6827/1305411055","heartbeat_back_addr":"172.21.15.196:6831/1305411055","heartbeat_front_addr":"172.21.15.196:6829/1305411055","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:21:04.088296+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:21:27.627518+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:21:50.679365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:22:14.402487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:22:35.262438+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:22:58.634666+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:23:22.347709+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:23:45.933082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/4034272136":"2024-04-15T15:19:45.647884+0000","172.21.15.16:0/506900340":"2024-04-15T15:19:45.647884+0000","172.21.15.16:6801/37325361":"2024-04-15T15:19:45.647884+0000","172.21.15.16:0/86456086":"2024-04-15T15:19:45.647884+0000","172.21.15.16:6800/37325361":"2024-04-15T15:19:45.647884+0000","172.21.15.16:6800/4006149639":"2024-04-15T15:19:22.735580+0000","172.21.15.16:0/646505488":"2024-04-15T15:19:22.735580+0000","172.21.15.16:0/1366879040":"2024-04-15T15:19:22.735580+0000","172.21.15.16:6801/4006149639":"2024-04-15T15:19:22.735580+0000","172.21.15.16:0/3638250575":"2024-04-15T15:19:22.735580+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T15:23:54.560 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T15:21:54.002288+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-14T15:23:54.560 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-14T15:23:54.785 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:23:54.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:54 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3656579057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:23:54.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:54 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3656579057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:23:55.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:54 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/3656579057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:23:55.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:55 smithi016 ceph-mon[24869]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-14T15:23:55.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:55 smithi016 ceph-mon[29517]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-14T15:23:56.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:55 smithi196 ceph-mon[25696]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-14T15:23:56.626 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-14T15:23:57.081 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:56 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/4036342615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-14T15:23:57.101 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:56 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4036342615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-14T15:23:57.101 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:56 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4036342615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-14T15:23:57.349 INFO:tasks.cephadm:Adding prometheus.a on smithi196 2024-04-14T15:23:57.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch apply prometheus '1;smithi196=a' 2024-04-14T15:23:57.976 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:57 smithi196 ceph-mon[25696]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-14T15:23:58.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:57 smithi016 ceph-mon[24869]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-14T15:23:58.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:57 smithi016 ceph-mon[29517]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-14T15:23:59.267 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled prometheus update... 2024-04-14T15:24:00.011 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:59 smithi196 ceph-mon[25696]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T15:24:00.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:24:00.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:24:00.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:24:00.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:24:00.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:24:00.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:23:59 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-14T15:24:00.035 DEBUG:teuthology.orchestra.run.smithi196:prometheus.a> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@prometheus.a.service 2024-04-14T15:24:00.038 INFO:tasks.cephadm:Adding node-exporter.a on smithi016 2024-04-14T15:24:00.038 INFO:tasks.cephadm:Adding node-exporter.b on smithi196 2024-04-14T15:24:00.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch apply node-exporter '2;smithi016=a;smithi196=b' 2024-04-14T15:24:00.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[29517]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T15:24:00.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:24:00.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:24:00.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:24:00.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:24:00.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:24:00.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-14T15:24:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[24869]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T15:24:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:24:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:24:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:24:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:24:00.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' 2024-04-14T15:24:00.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:23:59 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-14T15:24:00.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ignoring --setuser ceph since I am not root 2024-04-14T15:24:00.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ignoring --setgroup ceph since I am not root 2024-04-14T15:24:00.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-mgr[25103]: -- 172.21.15.16:0/1692351191 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5654e21da340 con 0x5654e21b3000 2024-04-14T15:24:00.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:00 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ignoring --setuser ceph since I am not root 2024-04-14T15:24:00.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:00 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ignoring --setgroup ceph since I am not root 2024-04-14T15:24:00.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:00 smithi196 ceph-mgr[26888]: -- 172.21.15.196:0/207213380 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557872826340 con 0x5578727fec00 2024-04-14T15:24:01.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:00 smithi196 ceph-mon[25696]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:01.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:00 smithi196 ceph-mon[25696]: Saving service prometheus spec with placement smithi196=a;count:1 2024-04-14T15:24:01.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:00 smithi196 ceph-mon[25696]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-14T15:24:01.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:00 smithi196 ceph-mon[25696]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-14T15:24:01.044 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:00 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:00.833+0000 7f01369b7000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:24:01.044 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:00 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:00.922+0000 7f01369b7000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:24:01.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:00.842+0000 7fd5a7e9b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:24:01.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:00.933+0000 7fd5a7e9b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:24:01.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-mon[29517]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:01.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-mon[29517]: Saving service prometheus spec with placement smithi196=a;count:1 2024-04-14T15:24:01.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-mon[29517]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-14T15:24:01.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-mon[29517]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-14T15:24:01.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-mon[24869]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:01.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-mon[24869]: Saving service prometheus spec with placement smithi196=a;count:1 2024-04-14T15:24:01.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-mon[24869]: from='mgr.14148 172.21.15.16:0/2784625236' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-14T15:24:01.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:00 smithi016 ceph-mon[24869]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-14T15:24:01.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:01.519+0000 7f01369b7000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:24:01.611 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:01.610+0000 7f01369b7000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:24:01.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:01.539+0000 7fd5a7e9b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:24:01.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:01.629+0000 7fd5a7e9b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:24:01.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:01.715+0000 7fd5a7e9b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:24:01.914 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:01.697+0000 7f01369b7000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:24:01.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:01.853+0000 7f01369b7000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:24:02.145 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:01.866+0000 7fd5a7e9b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:24:02.146 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:01.949+0000 7fd5a7e9b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:24:02.146 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:02.060+0000 7fd5a7e9b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:24:02.146 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:02.145+0000 7fd5a7e9b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:24:02.291 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:01.935+0000 7f01369b7000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:24:02.291 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:02.044+0000 7f01369b7000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:24:02.291 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:02.127+0000 7f01369b7000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:24:02.291 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:02.290+0000 7f01369b7000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:24:02.398 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:02.311+0000 7fd5a7e9b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:24:02.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:02.375+0000 7f01369b7000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:24:02.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:02.398+0000 7fd5a7e9b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:24:03.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:03.390+0000 7f01369b7000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:24:03.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:03.517+0000 7f01369b7000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:24:03.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:03 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:03.458+0000 7fd5a7e9b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:24:03.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:03 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:03.592+0000 7fd5a7e9b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:24:03.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:03 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:03.789+0000 7fd5a7e9b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:24:04.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:03 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:03.879+0000 7fd5a7e9b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:24:04.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:04.019+0000 7fd5a7e9b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:24:04.164 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:03.707+0000 7f01369b7000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:24:04.164 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:03.794+0000 7f01369b7000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:24:04.165 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:03.929+0000 7f01369b7000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:24:04.549 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:04.248+0000 7f01369b7000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:24:04.549 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:04.467+0000 7f01369b7000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:24:04.549 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:04.548+0000 7f01369b7000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:24:04.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:04.343+0000 7fd5a7e9b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:24:04.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:04.571+0000 7fd5a7e9b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:24:04.914 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:04.717+0000 7f01369b7000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:24:05.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:04.658+0000 7fd5a7e9b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:24:05.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:04.834+0000 7fd5a7e9b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:24:05.914 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:05.628+0000 7f01369b7000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:24:06.086 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:05 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:05.778+0000 7fd5a7e9b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:24:06.221 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:05.920+0000 7f01369b7000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:24:06.221 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:06.013+0000 7f01369b7000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:24:06.221 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:06.160+0000 7f01369b7000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:24:06.221 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: [14/Apr/2024:15:24:06] ENGINE Bus STARTING 2024-04-14T15:24:06.344 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:06.084+0000 7fd5a7e9b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:24:06.344 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:06.183+0000 7fd5a7e9b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:24:06.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[24869]: Standby manager daemon x restarted 2024-04-14T15:24:06.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[24869]: Standby manager daemon x started 2024-04-14T15:24:06.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:24:06.344 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:24:06.345 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:24:06.345 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:24:06.345 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[29517]: Standby manager daemon x restarted 2024-04-14T15:24:06.345 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[29517]: Standby manager daemon x started 2024-04-14T15:24:06.345 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:24:06.345 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:24:06.345 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:24:06.345 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:24:06.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:06 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:06.342+0000 7fd5a7e9b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:24:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-mon[25696]: Standby manager daemon x restarted 2024-04-14T15:24:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-mon[25696]: Standby manager daemon x started 2024-04-14T15:24:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:24:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:24:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:24:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/2629357344' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:24:06.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: CherryPy Checker: 2024-04-14T15:24:06.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: The Application mounted at '' has an empty config. 2024-04-14T15:24:06.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: 2024-04-14T15:24:06.666 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: [14/Apr/2024:15:24:06] ENGINE Serving on http://:::9283 2024-04-14T15:24:06.666 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: [14/Apr/2024:15:24:06] ENGINE Bus STARTED 2024-04-14T15:24:07.555 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:24:07] ENGINE Bus STARTING 2024-04-14T15:24:07.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-mon[24869]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-14T15:24:07.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-mon[24869]: Active manager daemon y restarted 2024-04-14T15:24:07.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-mon[24869]: Activating manager daemon y 2024-04-14T15:24:07.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-mon[24869]: osdmap e60: 8 total, 8 up, 8 in 2024-04-14T15:24:07.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-mon[29517]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-14T15:24:07.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-mon[29517]: Active manager daemon y restarted 2024-04-14T15:24:07.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-mon[29517]: Activating manager daemon y 2024-04-14T15:24:07.556 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-mon[29517]: osdmap e60: 8 total, 8 up, 8 in 2024-04-14T15:24:07.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:07 smithi196 ceph-mon[25696]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-14T15:24:07.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:07 smithi196 ceph-mon[25696]: Active manager daemon y restarted 2024-04-14T15:24:07.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:07 smithi196 ceph-mon[25696]: Activating manager daemon y 2024-04-14T15:24:07.614 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:07 smithi196 ceph-mon[25696]: osdmap e60: 8 total, 8 up, 8 in 2024-04-14T15:24:07.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: CherryPy Checker: 2024-04-14T15:24:07.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: The Application mounted at '' has an empty config. 2024-04-14T15:24:07.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:24:07.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:24:07] ENGINE Serving on http://:::9283 2024-04-14T15:24:07.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:24:07] ENGINE Bus STARTED 2024-04-14T15:24:07.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:24:07] ENGINE Bus STARTING 2024-04-14T15:24:08.239 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:24:07] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:24:08.239 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:07 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:24:07] ENGINE Bus STARTED 2024-04-14T15:24:08.247 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled node-exporter update... 2024-04-14T15:24:08.491 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: mgrmap e18: y(active, starting, since 0.885932s), standbys: x 2024-04-14T15:24:08.491 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:24:08.492 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: Manager daemon y is now available 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: mgrmap e18: y(active, starting, since 0.885932s), standbys: x 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:24:08.493 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: Manager daemon y is now available 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:24:08.494 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:24:08.495 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:08.495 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:24:08.495 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:24:08.495 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:08.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: mgrmap e18: y(active, starting, since 0.885932s), standbys: x 2024-04-14T15:24:08.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:24:08.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:24:08.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:24:08.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:24:08.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: Manager daemon y is now available 2024-04-14T15:24:08.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:24:08.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:24:08.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:24:08.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:24:08.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:08.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:24:08.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:24:08.603 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:08.943 DEBUG:teuthology.orchestra.run.smithi016:node-exporter.a> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.a.service 2024-04-14T15:24:08.946 DEBUG:teuthology.orchestra.run.smithi196:node-exporter.b> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.b.service 2024-04-14T15:24:08.948 INFO:tasks.cephadm:Adding alertmanager.a on smithi016 2024-04-14T15:24:08.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch apply alertmanager '1;smithi016=a' 2024-04-14T15:24:09.530 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[29517]: [14/Apr/2024:15:24:07] ENGINE Bus STARTING 2024-04-14T15:24:09.530 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[29517]: [14/Apr/2024:15:24:07] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:24:09.530 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[29517]: [14/Apr/2024:15:24:07] ENGINE Bus STARTED 2024-04-14T15:24:09.530 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[29517]: mgrmap e19: y(active, since 1.88958s), standbys: x 2024-04-14T15:24:09.530 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[29517]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi016=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:09.530 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[29517]: Saving service node-exporter spec with placement smithi016=a;smithi196=b;count:2 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[29517]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[24869]: [14/Apr/2024:15:24:07] ENGINE Bus STARTING 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[24869]: [14/Apr/2024:15:24:07] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[24869]: [14/Apr/2024:15:24:07] ENGINE Bus STARTED 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[24869]: mgrmap e19: y(active, since 1.88958s), standbys: x 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[24869]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi016=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[24869]: Saving service node-exporter spec with placement smithi016=a;smithi196=b;count:2 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[24869]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:09.531 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:09 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:09.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:09 smithi196 ceph-mon[25696]: [14/Apr/2024:15:24:07] ENGINE Bus STARTING 2024-04-14T15:24:09.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:09 smithi196 ceph-mon[25696]: [14/Apr/2024:15:24:07] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:24:09.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:09 smithi196 ceph-mon[25696]: [14/Apr/2024:15:24:07] ENGINE Bus STARTED 2024-04-14T15:24:09.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:09 smithi196 ceph-mon[25696]: mgrmap e19: y(active, since 1.88958s), standbys: x 2024-04-14T15:24:09.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:09 smithi196 ceph-mon[25696]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi016=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:09.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:09 smithi196 ceph-mon[25696]: Saving service node-exporter spec with placement smithi016=a;smithi196=b;count:2 2024-04-14T15:24:09.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:09 smithi196 ceph-mon[25696]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:09.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:09 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:09.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:09 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:10.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:10 smithi016 ceph-mon[29517]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:10.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:10 smithi016 ceph-mon[29517]: mgrmap e20: y(active, since 2s), standbys: x 2024-04-14T15:24:10.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:10 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:10.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:10 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:10.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:10 smithi016 ceph-mon[24869]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:10.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:10 smithi016 ceph-mon[24869]: mgrmap e20: y(active, since 2s), standbys: x 2024-04-14T15:24:10.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:10 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:10.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:10 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:10.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:10 smithi196 ceph-mon[25696]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:10.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:10 smithi196 ceph-mon[25696]: mgrmap e20: y(active, since 2s), standbys: x 2024-04-14T15:24:10.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:10 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:10.625 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:10 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:11.337 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled alertmanager update... 2024-04-14T15:24:11.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:11 smithi196 ceph-mon[25696]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:24:11.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:11 smithi196 ceph-mon[25696]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:24:11.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:11 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:11.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:11 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:11.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:11 smithi016 ceph-mon[29517]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:24:11.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:11 smithi016 ceph-mon[29517]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:24:11.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:11 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:11.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:11 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:11.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:11 smithi016 ceph-mon[24869]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:24:11.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:11 smithi016 ceph-mon[24869]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:24:11.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:11 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:11.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:11 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:12.118 DEBUG:teuthology.orchestra.run.smithi016:alertmanager.a> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@alertmanager.a.service 2024-04-14T15:24:12.120 INFO:tasks.cephadm:Adding grafana.a on smithi196 2024-04-14T15:24:12.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph orch apply grafana '1;smithi196=a' 2024-04-14T15:24:12.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:12 smithi196 ceph-mon[25696]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:24:12.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:12 smithi196 ceph-mon[25696]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:12.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:12 smithi196 ceph-mon[25696]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi016=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:12.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:12 smithi196 ceph-mon[25696]: Saving service alertmanager spec with placement smithi016=a;count:1 2024-04-14T15:24:12.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:12 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:12.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:12 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:12.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:12 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:12.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[29517]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:24:12.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[29517]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[29517]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi016=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[29517]: Saving service alertmanager spec with placement smithi016=a;count:1 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[24869]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[24869]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[24869]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi016=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[24869]: Saving service alertmanager spec with placement smithi016=a;count:1 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:12 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:13.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:13 smithi016 ceph-mon[29517]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:24:13.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:13 smithi016 ceph-mon[29517]: Deploying daemon prometheus.a on smithi196 2024-04-14T15:24:13.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:13 smithi016 ceph-mon[24869]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:24:13.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:13 smithi016 ceph-mon[24869]: Deploying daemon prometheus.a on smithi196 2024-04-14T15:24:13.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:13 smithi196 ceph-mon[25696]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:24:13.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:13 smithi196 ceph-mon[25696]: Deploying daemon prometheus.a on smithi196 2024-04-14T15:24:14.489 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled grafana update... 2024-04-14T15:24:14.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:14 smithi016 ceph-mon[24869]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:14.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:14 smithi016 ceph-mon[29517]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:14.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:14 smithi196 ceph-mon[25696]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:15.431 DEBUG:teuthology.orchestra.run.smithi196:grafana.a> sudo journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service 2024-04-14T15:24:15.434 INFO:tasks.cephadm:Setting up client nodes... 2024-04-14T15:24:15.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-14T15:24:15.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:15 smithi196 ceph-mon[25696]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:15.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:15 smithi196 ceph-mon[25696]: Saving service grafana spec with placement smithi196=a;count:1 2024-04-14T15:24:15.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:15 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:15.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:15 smithi196 ceph-mon[25696]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:15.753 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:15 smithi016 ceph-mon[29517]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:15.753 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:15 smithi016 ceph-mon[29517]: Saving service grafana spec with placement smithi196=a;count:1 2024-04-14T15:24:15.753 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:15 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:15.753 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:15 smithi016 ceph-mon[29517]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:15.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:15 smithi016 ceph-mon[24869]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:24:15.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:15 smithi016 ceph-mon[24869]: Saving service grafana spec with placement smithi196=a;count:1 2024-04-14T15:24:15.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:15 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:15.753 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:15 smithi016 ceph-mon[24869]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:17.484 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-04-14T15:24:17.484 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAh9RtmYdSlHBAAsLD+7iYfLw+b5ZY4jm6l3Q== 2024-04-14T15:24:18.172 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-14T15:24:18.172 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-14T15:24:18.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-14T15:24:18.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-14T15:24:18.536 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:18 smithi196 ceph-mon[25696]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:18.536 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:18 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2824059347' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:18.536 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:18 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:18.536 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:18 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:24:18.548 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:18 smithi016 ceph-mon[29517]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:18.548 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:18 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2824059347' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:18.548 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:18 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:18.548 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:18 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:24:18.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:18 smithi016 ceph-mon[24869]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:18.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:18 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2824059347' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:18.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:18 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:18.548 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:18 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:24:19.976 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:19 smithi196 ceph-mon[25696]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:20.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:19 smithi016 ceph-mon[29517]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:20.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:19 smithi016 ceph-mon[24869]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:22.204 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2024-04-14T15:24:22.205 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQAm9RtmARz5CxAAxCJt7pAB678AZY89pTqcIA== 2024-04-14T15:24:22.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:22 smithi016 ceph-mon[29517]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:22.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:22 smithi016 ceph-mon[29517]: from='client.? 172.21.15.196:0/2402406380' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:22.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:22 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:22.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:22 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:24:22.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:22 smithi016 ceph-mon[24869]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:22.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:22 smithi016 ceph-mon[24869]: from='client.? 172.21.15.196:0/2402406380' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:22.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:22 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:22.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:22 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:24:22.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:22 smithi196 ceph-mon[25696]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:22.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:22 smithi196 ceph-mon[25696]: from='client.? 172.21.15.196:0/2402406380' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:22.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:22 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T15:24:22.657 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:22 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T15:24:24.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:24 smithi016 ceph-mon[24869]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:24.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:24 smithi016 ceph-mon[29517]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:24.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:24 smithi196 ceph-mon[25696]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:24.922 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-04-14T15:24:24.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-14T15:24:24.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-14T15:24:24.969 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-14T15:24:24.969 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-14T15:24:24.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph mgr dump --format=json 2024-04-14T15:24:25.198 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:25.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:25 smithi196 systemd[1]: Starting Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:24:26.284 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 podman[40834]: 2024-04-14 15:24:25.921736745 +0000 UTC m=+0.018878703 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-14T15:24:26.285 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 podman[40834]: 2024-04-14 15:24:26.02386138 +0000 UTC m=+0.121003356 container create 7d73a9ee43c2af939ada1adb8e3ca15cca69532df02b9c762a984d93bbb8ada2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:24:26.285 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 podman[40834]: 2024-04-14 15:24:26.259362719 +0000 UTC m=+0.356504649 container init 7d73a9ee43c2af939ada1adb8e3ca15cca69532df02b9c762a984d93bbb8ada2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:24:26.285 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 podman[40834]: 2024-04-14 15:24:26.262041549 +0000 UTC m=+0.359183480 container start 7d73a9ee43c2af939ada1adb8e3ca15cca69532df02b9c762a984d93bbb8ada2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:24:26.563 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:26 smithi016 ceph-mon[29517]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:26.563 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:26 smithi016 ceph-mon[24869]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:26.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-mon[25696]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:26.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.284Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-14T15:24:26.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.285Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-14T15:24:26.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.285Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-14T15:24:26.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.285Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi196 (none))" 2024-04-14T15:24:26.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.285Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T15:24:26.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.285Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T15:24:26.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.286Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T15:24:26.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.287Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-14T15:24:26.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.287Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-14T15:24:26.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.290Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T15:24:26.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.290Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.722µs 2024-04-14T15:24:26.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.290Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T15:24:26.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.290Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-14T15:24:26.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.290Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.857µs wal_replay_duration=133.376µs total_replay_duration=167.397µs 2024-04-14T15:24:26.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.291Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T15:24:26.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.291Z caller=main.go:947 level=info msg="TSDB started" 2024-04-14T15:24:26.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.291Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T15:24:26.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.305Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.344734ms db_storage=568ns remote_storage=1.2µs web_handler=303ns query_engine=599ns scrape=479.426µs scrape_sd=15.908µs notify=1.046µs notify_sd=2.021µs rules=13.65797ms 2024-04-14T15:24:26.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:24:26.305Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-14T15:24:26.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 bash[40834]: 7d73a9ee43c2af939ada1adb8e3ca15cca69532df02b9c762a984d93bbb8ada2 2024-04-14T15:24:26.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:24:26 smithi196 systemd[1]: Started Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:24:27.081 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:24:27.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:27 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:27 smithi016 ceph-mon[24869]: Deploying daemon node-exporter.a on smithi016 2024-04-14T15:24:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:27 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3296747759' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T15:24:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:27 smithi016 ceph-mon[24869]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:27 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:27.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:27 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:27.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:27 smithi016 ceph-mon[29517]: Deploying daemon node-exporter.a on smithi016 2024-04-14T15:24:27.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:27 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3296747759' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T15:24:27.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:27 smithi016 ceph-mon[29517]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:27.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:27 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:27.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:27 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:27 smithi196 ceph-mon[25696]: Deploying daemon node-exporter.a on smithi016 2024-04-14T15:24:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:27 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/3296747759' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T15:24:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:27 smithi196 ceph-mon[25696]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:27 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:27.811 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"active_gid":24349,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":3043941881},{"type":"v1","addr":"172.21.15.16:6801","nonce":3043941881}]},"active_addr":"172.21.15.16:6801/3043941881","active_change":"2024-04-14T15:24:06.346888+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34346,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/","prometheus":"http://172.21.15.16:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":42332771}]},{"addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2330461653}]},{"addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1935194990}]},{"addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2858662230}]}]}} 2024-04-14T15:24:27.814 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-14T15:24:27.814 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-14T15:24:27.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T15:24:28.053 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:29.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:29 smithi016 ceph-mon[24869]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:29.936 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:29 smithi016 ceph-mon[29517]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:30.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:29 smithi196 ceph-mon[25696]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:30.521 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:24:30.521 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":60,"fsid":"3ca5917a-fa72-11ee-bc8e-c7b262605968","created":"2024-04-14T15:18:59.224477+0000","modified":"2024-04-14T15:24:06.346485+0000","last_up_change":"2024-04-14T15:23:47.838098+0000","last_in_change":"2024-04-14T15:23:30.452634+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T15:21:54.002288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b8767d4c-2155-4a83-8522-ac3042eaa0be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6803","nonce":696303661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6805","nonce":696303661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6809","nonce":696303661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6807","nonce":696303661}]},"public_addr":"172.21.15.16:6803/696303661","cluster_addr":"172.21.15.16:6805/696303661","heartbeat_back_addr":"172.21.15.16:6809/696303661","heartbeat_front_addr":"172.21.15.16:6807/696303661","state":["exists","up"]},{"osd":1,"uuid":"ef5e2c3d-9d71-41c3-a640-188d71c4fe72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6811","nonce":1936311739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6813","nonce":1936311739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6817","nonce":1936311739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6815","nonce":1936311739}]},"public_addr":"172.21.15.16:6811/1936311739","cluster_addr":"172.21.15.16:6813/1936311739","heartbeat_back_addr":"172.21.15.16:6817/1936311739","heartbeat_front_addr":"172.21.15.16:6815/1936311739","state":["exists","up"]},{"osd":2,"uuid":"0b1e7e85-29f9-4b4f-a16c-d80ed36951c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6819","nonce":701104650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6821","nonce":701104650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6825","nonce":701104650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6823","nonce":701104650}]},"public_addr":"172.21.15.16:6819/701104650","cluster_addr":"172.21.15.16:6821/701104650","heartbeat_back_addr":"172.21.15.16:6825/701104650","heartbeat_front_addr":"172.21.15.16:6823/701104650","state":["exists","up"]},{"osd":3,"uuid":"5f943ae4-952d-4e12-97c5-2a23c47a1c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6827","nonce":3777247940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6829","nonce":3777247940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6833","nonce":3777247940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6831","nonce":3777247940}]},"public_addr":"172.21.15.16:6827/3777247940","cluster_addr":"172.21.15.16:6829/3777247940","heartbeat_back_addr":"172.21.15.16:6833/3777247940","heartbeat_front_addr":"172.21.15.16:6831/3777247940","state":["exists","up"]},{"osd":4,"uuid":"1ad25b86-55ee-4769-991e-de2a723b3516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6801","nonce":61582609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6803","nonce":61582609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6807","nonce":61582609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6805","nonce":61582609}]},"public_addr":"172.21.15.196:6801/61582609","cluster_addr":"172.21.15.196:6803/61582609","heartbeat_back_addr":"172.21.15.196:6807/61582609","heartbeat_front_addr":"172.21.15.196:6805/61582609","state":["exists","up"]},{"osd":5,"uuid":"185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6809","nonce":786436243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6811","nonce":786436243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6815","nonce":786436243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6813","nonce":786436243}]},"public_addr":"172.21.15.196:6809/786436243","cluster_addr":"172.21.15.196:6811/786436243","heartbeat_back_addr":"172.21.15.196:6815/786436243","heartbeat_front_addr":"172.21.15.196:6813/786436243","state":["exists","up"]},{"osd":6,"uuid":"175090a7-f054-4e4d-8c65-d9b503e6c547","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6817","nonce":3846591221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6819","nonce":3846591221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6823","nonce":3846591221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6821","nonce":3846591221}]},"public_addr":"172.21.15.196:6817/3846591221","cluster_addr":"172.21.15.196:6819/3846591221","heartbeat_back_addr":"172.21.15.196:6823/3846591221","heartbeat_front_addr":"172.21.15.196:6821/3846591221","state":["exists","up"]},{"osd":7,"uuid":"2161a137-f1f6-473c-93d2-39ede9f5d6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6825","nonce":1305411055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6827","nonce":1305411055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6831","nonce":1305411055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6829","nonce":1305411055}]},"public_addr":"172.21.15.196:6825/1305411055","cluster_addr":"172.21.15.196:6827/1305411055","heartbeat_back_addr":"172.21.15.196:6831/1305411055","heartbeat_front_addr":"172.21.15.196:6829/1305411055","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:21:04.088296+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:21:27.627518+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:21:50.679365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:22:14.402487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:22:35.262438+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:22:58.634666+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:23:22.347709+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:23:45.933082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6801/2467795236":"2024-04-15T15:24:06.346435+0000","172.21.15.16:0/2899000330":"2024-04-15T15:24:06.346435+0000","172.21.15.16:0/304800256":"2024-04-15T15:24:06.346435+0000","172.21.15.16:0/4034272136":"2024-04-15T15:19:45.647884+0000","172.21.15.16:0/506900340":"2024-04-15T15:19:45.647884+0000","172.21.15.16:6801/37325361":"2024-04-15T15:19:45.647884+0000","172.21.15.16:0/86456086":"2024-04-15T15:19:45.647884+0000","172.21.15.16:6800/2467795236":"2024-04-15T15:24:06.346435+0000","172.21.15.16:6800/37325361":"2024-04-15T15:19:45.647884+0000","172.21.15.16:6800/4006149639":"2024-04-15T15:19:22.735580+0000","172.21.15.16:0/910690549":"2024-04-15T15:24:06.346435+0000","172.21.15.16:0/646505488":"2024-04-15T15:19:22.735580+0000","172.21.15.16:0/2585190683":"2024-04-15T15:24:06.346435+0000","172.21.15.16:0/1366879040":"2024-04-15T15:19:22.735580+0000","172.21.15.16:6801/4006149639":"2024-04-15T15:19:22.735580+0000","172.21.15.16:0/3638250575":"2024-04-15T15:19:22.735580+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T15:24:30.824 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:30 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/95725398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:24:30.824 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:30 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/95725398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:24:31.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:30 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/95725398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:24:31.610 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:31 smithi016 bash[44400]: Writing manifest to image destination 2024-04-14T15:24:31.719 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-14T15:24:31.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T15:24:31.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:31 smithi016 ceph-mon[29517]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:31.924 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:31 smithi016 ceph-mon[24869]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:31.924 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:31 smithi016 podman[44400]: 2024-04-14 15:24:31.567960374 +0000 UTC m=+3.742477532 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-14T15:24:31.924 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:31 smithi016 podman[44400]: 2024-04-14 15:24:31.817198387 +0000 UTC m=+3.991715552 container create 568fff30d221b6f3b504625492b055ce914c01f72460488b86f07a6cee67394b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:24:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:32.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:31 smithi196 ceph-mon[25696]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:32.279 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 podman[44400]: 2024-04-14 15:24:32.01936512 +0000 UTC m=+4.193882278 container init 568fff30d221b6f3b504625492b055ce914c01f72460488b86f07a6cee67394b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:24:32.279 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 podman[44400]: 2024-04-14 15:24:32.022182792 +0000 UTC m=+4.196699953 container start 568fff30d221b6f3b504625492b055ce914c01f72460488b86f07a6cee67394b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:24:32.279 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.025Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-14T15:24:32.279 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.026Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-14T15:24:32.279 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.026Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-14T15:24:32.279 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.026Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T15:24:32.279 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=arp 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=edac 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-14T15:24:32.280 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=os 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-14T15:24:32.281 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=stat 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=time 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=uname 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[44629]: ts=2024-04-14T15:24:32.027Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-14T15:24:32.282 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 bash[44400]: 568fff30d221b6f3b504625492b055ce914c01f72460488b86f07a6cee67394b 2024-04-14T15:24:32.283 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:24:32 smithi016 systemd[1]: Started Ceph node-exporter.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:24:32.589 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:24:32] "GET /metrics HTTP/1.1" 200 191505 "" "Prometheus/2.33.4" 2024-04-14T15:24:32.835 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:32 smithi196 systemd[1]: Starting Ceph node-exporter.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:24:33.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:33 smithi196 bash[41085]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-14T15:24:33.331 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:33 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:33.331 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:33 smithi016 ceph-mon[29517]: Deploying daemon node-exporter.b on smithi196 2024-04-14T15:24:33.331 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:33 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:33.332 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:33 smithi016 ceph-mon[24869]: Deploying daemon node-exporter.b on smithi196 2024-04-14T15:24:33.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:33 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:33.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:33 smithi196 ceph-mon[25696]: Deploying daemon node-exporter.b on smithi196 2024-04-14T15:24:34.097 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:24:34.098 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":60,"fsid":"3ca5917a-fa72-11ee-bc8e-c7b262605968","created":"2024-04-14T15:18:59.224477+0000","modified":"2024-04-14T15:24:06.346485+0000","last_up_change":"2024-04-14T15:23:47.838098+0000","last_in_change":"2024-04-14T15:23:30.452634+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T15:21:54.002288+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b8767d4c-2155-4a83-8522-ac3042eaa0be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6803","nonce":696303661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6805","nonce":696303661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6809","nonce":696303661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":696303661},{"type":"v1","addr":"172.21.15.16:6807","nonce":696303661}]},"public_addr":"172.21.15.16:6803/696303661","cluster_addr":"172.21.15.16:6805/696303661","heartbeat_back_addr":"172.21.15.16:6809/696303661","heartbeat_front_addr":"172.21.15.16:6807/696303661","state":["exists","up"]},{"osd":1,"uuid":"ef5e2c3d-9d71-41c3-a640-188d71c4fe72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6811","nonce":1936311739}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6813","nonce":1936311739}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6817","nonce":1936311739}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":1936311739},{"type":"v1","addr":"172.21.15.16:6815","nonce":1936311739}]},"public_addr":"172.21.15.16:6811/1936311739","cluster_addr":"172.21.15.16:6813/1936311739","heartbeat_back_addr":"172.21.15.16:6817/1936311739","heartbeat_front_addr":"172.21.15.16:6815/1936311739","state":["exists","up"]},{"osd":2,"uuid":"0b1e7e85-29f9-4b4f-a16c-d80ed36951c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6819","nonce":701104650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6821","nonce":701104650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6825","nonce":701104650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":701104650},{"type":"v1","addr":"172.21.15.16:6823","nonce":701104650}]},"public_addr":"172.21.15.16:6819/701104650","cluster_addr":"172.21.15.16:6821/701104650","heartbeat_back_addr":"172.21.15.16:6825/701104650","heartbeat_front_addr":"172.21.15.16:6823/701104650","state":["exists","up"]},{"osd":3,"uuid":"5f943ae4-952d-4e12-97c5-2a23c47a1c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6827","nonce":3777247940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6829","nonce":3777247940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6833","nonce":3777247940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":3777247940},{"type":"v1","addr":"172.21.15.16:6831","nonce":3777247940}]},"public_addr":"172.21.15.16:6827/3777247940","cluster_addr":"172.21.15.16:6829/3777247940","heartbeat_back_addr":"172.21.15.16:6833/3777247940","heartbeat_front_addr":"172.21.15.16:6831/3777247940","state":["exists","up"]},{"osd":4,"uuid":"1ad25b86-55ee-4769-991e-de2a723b3516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6801","nonce":61582609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6803","nonce":61582609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6807","nonce":61582609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":61582609},{"type":"v1","addr":"172.21.15.196:6805","nonce":61582609}]},"public_addr":"172.21.15.196:6801/61582609","cluster_addr":"172.21.15.196:6803/61582609","heartbeat_back_addr":"172.21.15.196:6807/61582609","heartbeat_front_addr":"172.21.15.196:6805/61582609","state":["exists","up"]},{"osd":5,"uuid":"185c46ea-ccda-4ce2-82d0-6baf9cb0ad9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6809","nonce":786436243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6811","nonce":786436243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6815","nonce":786436243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":786436243},{"type":"v1","addr":"172.21.15.196:6813","nonce":786436243}]},"public_addr":"172.21.15.196:6809/786436243","cluster_addr":"172.21.15.196:6811/786436243","heartbeat_back_addr":"172.21.15.196:6815/786436243","heartbeat_front_addr":"172.21.15.196:6813/786436243","state":["exists","up"]},{"osd":6,"uuid":"175090a7-f054-4e4d-8c65-d9b503e6c547","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6817","nonce":3846591221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6819","nonce":3846591221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6823","nonce":3846591221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3846591221},{"type":"v1","addr":"172.21.15.196:6821","nonce":3846591221}]},"public_addr":"172.21.15.196:6817/3846591221","cluster_addr":"172.21.15.196:6819/3846591221","heartbeat_back_addr":"172.21.15.196:6823/3846591221","heartbeat_front_addr":"172.21.15.196:6821/3846591221","state":["exists","up"]},{"osd":7,"uuid":"2161a137-f1f6-473c-93d2-39ede9f5d6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6825","nonce":1305411055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6827","nonce":1305411055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6831","nonce":1305411055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":1305411055},{"type":"v1","addr":"172.21.15.196:6829","nonce":1305411055}]},"public_addr":"172.21.15.196:6825/1305411055","cluster_addr":"172.21.15.196:6827/1305411055","heartbeat_back_addr":"172.21.15.196:6831/1305411055","heartbeat_front_addr":"172.21.15.196:6829/1305411055","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:21:04.088296+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:21:27.627518+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:21:50.679365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:22:14.402487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:22:35.262438+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:22:58.634666+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:23:22.347709+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T15:23:45.933082+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6801/2467795236":"2024-04-15T15:24:06.346435+0000","172.21.15.16:0/2899000330":"2024-04-15T15:24:06.346435+0000","172.21.15.16:0/304800256":"2024-04-15T15:24:06.346435+0000","172.21.15.16:0/4034272136":"2024-04-15T15:19:45.647884+0000","172.21.15.16:0/506900340":"2024-04-15T15:19:45.647884+0000","172.21.15.16:6801/37325361":"2024-04-15T15:19:45.647884+0000","172.21.15.16:0/86456086":"2024-04-15T15:19:45.647884+0000","172.21.15.16:6800/2467795236":"2024-04-15T15:24:06.346435+0000","172.21.15.16:6800/37325361":"2024-04-15T15:19:45.647884+0000","172.21.15.16:6800/4006149639":"2024-04-15T15:19:22.735580+0000","172.21.15.16:0/910690549":"2024-04-15T15:24:06.346435+0000","172.21.15.16:0/646505488":"2024-04-15T15:19:22.735580+0000","172.21.15.16:0/2585190683":"2024-04-15T15:24:06.346435+0000","172.21.15.16:0/1366879040":"2024-04-15T15:19:22.735580+0000","172.21.15.16:6801/4006149639":"2024-04-15T15:19:22.735580+0000","172.21.15.16:0/3638250575":"2024-04-15T15:19:22.735580+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T15:24:34.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:33 smithi196 bash[41085]: Getting image source signatures 2024-04-14T15:24:34.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:33 smithi196 bash[41085]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-14T15:24:34.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:33 smithi196 bash[41085]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-14T15:24:34.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:33 smithi196 bash[41085]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-14T15:24:34.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:34 smithi016 ceph-mon[24869]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:34.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:34 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/531467100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:24:34.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:34 smithi016 ceph-mon[29517]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:34.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:34 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/531467100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:24:34.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:34 smithi196 ceph-mon[25696]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:34.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:34 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/531467100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T15:24:34.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-14T15:24:34.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-14T15:24:34.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-14T15:24:34.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-14T15:24:34.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-14T15:24:34.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-14T15:24:34.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-14T15:24:34.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-14T15:24:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:35.202 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:35.203 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:35.212 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:35.215 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:35.216 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:36.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:36 smithi016 ceph-mon[29517]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:36.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:36 smithi016 ceph-mon[24869]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:36.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:36 smithi196 ceph-mon[25696]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:37.415 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 bash[41085]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-14T15:24:37.415 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 bash[41085]: Writing manifest to image destination 2024-04-14T15:24:37.915 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 podman[41085]: 2024-04-14 15:24:37.595372354 +0000 UTC m=+4.660233186 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-14T15:24:37.915 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 podman[41085]: 2024-04-14 15:24:37.710513928 +0000 UTC m=+4.775374760 container create 142e5a468751044dade65065274010ce8ae8b745f4d909cdac85b8ddc7e25aa2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:24:38.303 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 podman[41085]: 2024-04-14 15:24:37.938065252 +0000 UTC m=+5.002926094 container init 142e5a468751044dade65065274010ce8ae8b745f4d909cdac85b8ddc7e25aa2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:24:38.303 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 podman[41085]: 2024-04-14 15:24:37.941010211 +0000 UTC m=+5.005871044 container start 142e5a468751044dade65065274010ce8ae8b745f4d909cdac85b8ddc7e25aa2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:24:38.303 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.945Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-14T15:24:38.303 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.945Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-14T15:24:38.303 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.945Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-14T15:24:38.303 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.945Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T15:24:38.303 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-14T15:24:38.303 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=arp 2024-04-14T15:24:38.303 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-14T15:24:38.303 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-14T15:24:38.303 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=edac 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-14T15:24:38.304 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=os 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=stat 2024-04-14T15:24:38.305 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=time 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=uname 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[41134]: ts=2024-04-14T15:24:37.946Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:38 smithi196 bash[41085]: 142e5a468751044dade65065274010ce8ae8b745f4d909cdac85b8ddc7e25aa2 2024-04-14T15:24:38.306 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:24:38 smithi196 systemd[1]: Started Ceph node-exporter.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:24:38.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:38 smithi016 ceph-mon[24869]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:38.426 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:38 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:38.426 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:38 smithi016 ceph-mon[29517]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:38.426 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:38 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:38.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:38 smithi196 ceph-mon[25696]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:38.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:38 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:39.602 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:39 smithi016 ceph-mon[29517]: Deploying daemon alertmanager.a on smithi016 2024-04-14T15:24:39.602 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:39 smithi016 ceph-mon[24869]: Deploying daemon alertmanager.a on smithi016 2024-04-14T15:24:39.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:39 smithi196 ceph-mon[25696]: Deploying daemon alertmanager.a on smithi016 2024-04-14T15:24:39.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:39 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:24:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:24:40.608 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:40 smithi016 ceph-mon[29517]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:40.608 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:40 smithi016 ceph-mon[24869]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:40.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:40 smithi196 ceph-mon[25696]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:42.395 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:24:42] "GET /metrics HTTP/1.1" 200 191485 "" "Prometheus/2.33.4" 2024-04-14T15:24:42.395 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:42 smithi016 ceph-mon[29517]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:42.395 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:42 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:42.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:42 smithi016 ceph-mon[24869]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:42.395 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:42 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:42 smithi196 ceph-mon[25696]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:42 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:24:44.451 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:44 smithi016 ceph-mon[29517]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:44.452 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:44 smithi016 ceph-mon[24869]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:44.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:44 smithi196 ceph-mon[25696]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:46.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:46 smithi016 ceph-mon[29517]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:46.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:46 smithi016 ceph-mon[24869]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:46.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:46 smithi196 ceph-mon[25696]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:48.120 INFO:teuthology.orchestra.run.smithi016.stdout:77309411365 2024-04-14T15:24:48.120 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-14T15:24:48.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:48 smithi016 ceph-mon[24869]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:48.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:48 smithi016 ceph-mon[29517]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:48.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:48 smithi196 ceph-mon[25696]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:49.122 INFO:teuthology.orchestra.run.smithi016.stdout:34359738414 2024-04-14T15:24:49.122 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-14T15:24:49.123 INFO:teuthology.orchestra.run.smithi016.stdout:133143986204 2024-04-14T15:24:49.123 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-14T15:24:49.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:49 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:24:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:24:50.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:49 smithi016 ceph-mon[29517]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:49 smithi016 ceph-mon[24869]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:50.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:49 smithi196 ceph-mon[25696]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:50.666 INFO:teuthology.orchestra.run.smithi016.stdout:55834574889 2024-04-14T15:24:50.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-14T15:24:52.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:52 smithi016 ceph-mon[24869]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:52 smithi016 ceph-mon[29517]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:52.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:24:52 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:24:52] "GET /metrics HTTP/1.1" 200 191478 "" "Prometheus/2.33.4" 2024-04-14T15:24:52.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:52 smithi196 ceph-mon[25696]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:52.802 INFO:teuthology.orchestra.run.smithi016.stdout:201863462930 2024-04-14T15:24:52.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-14T15:24:52.917 INFO:teuthology.orchestra.run.smithi016.stdout:236223201291 2024-04-14T15:24:52.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-14T15:24:54.580 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:54 smithi016 ceph-mon[29517]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:54.580 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:54 smithi016 ceph-mon[24869]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:54 smithi196 ceph-mon[25696]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:55.320 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:55.321 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:55.321 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:55.322 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:56.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:56 smithi016 ceph-mon[24869]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:56.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:56 smithi016 ceph-mon[29517]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:56.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:56 smithi196 ceph-mon[25696]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:56.939 INFO:teuthology.orchestra.run.smithi016.stdout:167503724568 2024-04-14T15:24:56.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-14T15:24:57.202 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:24:57.614 INFO:teuthology.orchestra.run.smithi016.stdout:111669149729 2024-04-14T15:24:57.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-14T15:24:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:58 smithi016 ceph-mon[29517]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:58 smithi016 ceph-mon[24869]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:58 smithi196 ceph-mon[25696]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:59.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:24:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:24:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:24:59.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:24:59 smithi016 ceph-mon[24869]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:59.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:24:59 smithi016 ceph-mon[29517]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:24:59.872 INFO:teuthology.orchestra.run.smithi016.stdout:77309411368 2024-04-14T15:25:00.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:24:59 smithi196 ceph-mon[25696]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:00.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:00 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/270797329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T15:25:00.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:00 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/270797329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T15:25:01.085 INFO:teuthology.orchestra.run.smithi016.stdout:34359738417 2024-04-14T15:25:01.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:00 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/270797329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T15:25:01.779 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:01 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2916774718' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T15:25:01.779 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:01 smithi016 ceph-mon[29517]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:01.779 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:01 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2916774718' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T15:25:01.780 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:01 smithi016 ceph-mon[24869]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:01.973 INFO:teuthology.orchestra.run.smithi016.stdout:201863462933 2024-04-14T15:25:02.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:01 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2916774718' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T15:25:02.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:01 smithi196 ceph-mon[25696]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:02.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:25:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:25:02] "GET /metrics HTTP/1.1" 200 191478 "" "Prometheus/2.33.4" 2024-04-14T15:25:02.736 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738417 for osd.0 2024-04-14T15:25:02.736 DEBUG:teuthology.parallel:result is None 2024-04-14T15:25:03.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:02 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/614845652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T15:25:03.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:02 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/614845652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T15:25:03.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:02 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/614845652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T15:25:03.233 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462930 got 201863462933 for osd.6 2024-04-14T15:25:03.233 DEBUG:teuthology.parallel:result is None 2024-04-14T15:25:03.428 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:03.993 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:03 smithi016 ceph-mon[24869]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:03.993 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:03 smithi016 ceph-mon[29517]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:03.993 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:03 smithi016 podman[46854]: 2024-04-14 15:25:03.993185826 +0000 UTC m=+2.616448531 container create dd21ba7bfc250dcfc17883937b90bf924a717bda10f1c79d17a5e340ab27920b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:04.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:03 smithi196 ceph-mon[25696]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:04.244 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 podman[46854]: 2024-04-14 15:25:04.204068169 +0000 UTC m=+2.827330812 container init dd21ba7bfc250dcfc17883937b90bf924a717bda10f1c79d17a5e340ab27920b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:04.245 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 podman[46854]: 2024-04-14 15:25:04.206722223 +0000 UTC m=+2.829984869 container start dd21ba7bfc250dcfc17883937b90bf924a717bda10f1c79d17a5e340ab27920b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:04.245 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[46954]: level=info ts=2024-04-14T15:25:04.217Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-14T15:25:04.245 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[46954]: level=info ts=2024-04-14T15:25:04.217Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-14T15:25:04.245 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[46954]: level=info ts=2024-04-14T15:25:04.218Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.16 port=9094 2024-04-14T15:25:04.245 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[46954]: level=info ts=2024-04-14T15:25:04.219Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-14T15:25:04.610 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[46954]: level=info ts=2024-04-14T15:25:04.244Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T15:25:04.610 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[46954]: level=info ts=2024-04-14T15:25:04.245Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T15:25:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[46954]: level=info ts=2024-04-14T15:25:04.247Z caller=main.go:518 msg=Listening address=:9093 2024-04-14T15:25:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[46954]: level=info ts=2024-04-14T15:25:04.247Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-14T15:25:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 bash[46854]: dd21ba7bfc250dcfc17883937b90bf924a717bda10f1c79d17a5e340ab27920b 2024-04-14T15:25:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:04 smithi016 systemd[1]: Started Ceph alertmanager.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:25:04.761 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411368 for osd.2 2024-04-14T15:25:04.762 DEBUG:teuthology.parallel:result is None 2024-04-14T15:25:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:05.476 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:05 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:05.476 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:05 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:25:05.476 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:05 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:05.476 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:05 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:05.476 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:05 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:05.477 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:05 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:25:05.477 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:05 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:05.477 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:05 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:05.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:05 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:05.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:05 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:25:05.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:05 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:05.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:05 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:06.272 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:06 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[46954]: level=info ts=2024-04-14T15:25:06.219Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000132191s 2024-04-14T15:25:06.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:06 smithi196 ceph-mon[25696]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:06.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:06 smithi016 ceph-mon[29517]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:06.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:06 smithi016 ceph-mon[24869]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:07.811 INFO:teuthology.orchestra.run.smithi016.stdout:111669149732 2024-04-14T15:25:08.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:08.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[29517]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:08.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:08.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:25:08.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:25:08.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/177223961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T15:25:08.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:08.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[24869]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:08.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:08.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:25:08.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:25:08.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:07 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/177223961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T15:25:08.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:08.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:07 smithi196 ceph-mon[25696]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:08.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:08.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:25:08.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:25:08.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:07 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/177223961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T15:25:08.526 INFO:teuthology.orchestra.run.smithi016.stdout:236223201296 2024-04-14T15:25:09.160 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201291 got 236223201296 for osd.7 2024-04-14T15:25:09.161 DEBUG:teuthology.parallel:result is None 2024-04-14T15:25:09.254 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:08 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/646496095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T15:25:09.254 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:08 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/646496095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T15:25:09.280 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149729 got 111669149732 for osd.3 2024-04-14T15:25:09.280 DEBUG:teuthology.parallel:result is None 2024-04-14T15:25:09.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:08 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/646496095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T15:25:09.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:25:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:25:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:25:09.869 INFO:teuthology.orchestra.run.smithi016.stdout:55834574894 2024-04-14T15:25:10.156 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:09 smithi016 ceph-mon[29517]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:10.156 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:09 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4148753603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T15:25:10.156 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:09 smithi016 ceph-mon[24869]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:10.156 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:09 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4148753603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T15:25:10.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:09 smithi196 ceph-mon[25696]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:09 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/4148753603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T15:25:10.749 INFO:teuthology.orchestra.run.smithi016.stdout:167503724572 2024-04-14T15:25:11.068 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:10 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1873535307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T15:25:11.068 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:10 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1873535307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T15:25:11.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:10 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1873535307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T15:25:11.473 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724568 got 167503724572 for osd.5 2024-04-14T15:25:11.474 DEBUG:teuthology.parallel:result is None 2024-04-14T15:25:12.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:11 smithi016 ceph-mon[29517]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:12.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:11 smithi016 ceph-mon[24869]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:12.187 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574894 for osd.1 2024-04-14T15:25:12.187 DEBUG:teuthology.parallel:result is None 2024-04-14T15:25:12.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:11 smithi196 ceph-mon[25696]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:12.449 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:25:12 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:25:12] "GET /metrics HTTP/1.1" 200 191489 "" "Prometheus/2.33.4" 2024-04-14T15:25:13.504 INFO:teuthology.orchestra.run.smithi016.stdout:133143986210 2024-04-14T15:25:14.045 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986204 got 133143986210 for osd.4 2024-04-14T15:25:14.045 DEBUG:teuthology.parallel:result is None 2024-04-14T15:25:14.045 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-14T15:25:14.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:14 smithi016 ceph-mon[29517]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:14.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:14 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1922761554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T15:25:14.361 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:14 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[46954]: level=info ts=2024-04-14T15:25:14.222Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002321458s 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:14 smithi016 ceph-mon[24869]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:14.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:14 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1922761554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T15:25:14.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:14 smithi196 ceph-mon[25696]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:14.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:14 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1922761554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T15:25:14.714 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:16.474 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:16 smithi016 ceph-mon[29517]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:16.474 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:16 smithi016 ceph-mon[24869]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:16.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:16 smithi196 ceph-mon[25696]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:17.143 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:25:17.145 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-14T15:25:17.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:25:17.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[24869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:25:17.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[24869]: Deploying daemon grafana.a on smithi196 2024-04-14T15:25:17.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[24869]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:25:17.770 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[24869]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:17.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:25:17.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[29517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:25:17.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[29517]: Deploying daemon grafana.a on smithi196 2024-04-14T15:25:17.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[29517]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:25:17.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:17 smithi016 ceph-mon[29517]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:17.794 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-14T15:25:15.251556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48980,"kb_used_data":6164,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912236,"statfs":{"total":767960285184,"available":767910129664,"internally_reserved":0,"allocated":6311936,"data_stored":3511317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002258"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":263,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-14T15:24:07.323617+0000","last_change":"2024-04-14T15:23:51.916120+0000","last_active":"2024-04-14T15:24:07.323617+0000","last_peered":"2024-04-14T15:24:07.323617+0000","last_clean":"2024-04-14T15:24:07.323617+0000","last_became_active":"2024-04-14T15:23:51.915586+0000","last_became_peered":"2024-04-14T15:23:51.915586+0000","last_unstale":"2024-04-14T15:24:07.323617+0000","last_undegraded":"2024-04-14T15:24:07.323617+0000","last_fullsized":"2024-04-14T15:24:07.323617+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:54.677147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:54.677147+0000","last_clean_scrub_stamp":"2024-04-14T15:21:54.677147+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:57:46.478125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.44700000000000001}]},{"osd":1,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.52300000000000002}]},{"osd":2,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.59299999999999997}]},{"osd":3,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.63400000000000001}]},{"osd":4,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46400000000000002}]},{"osd":5,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.505},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.621}]},{"osd":6,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.60499999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6160,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738992,"statfs":{"total":95995035648,"available":95988727808,"internally_reserved":0,"allocated":1122304,"data_stored":769535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.51600000000000001}]},{"osd":1,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.45200000000000001}]},{"osd":2,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.48999999999999999}]},{"osd":3,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.58499999999999996}]},{"osd":5,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.59599999999999997}]},{"osd":7,"last update":"Sun Apr 14 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.56999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.35699999999999998,"15min":0.33700000000000002},"min":{"1min":0.24399999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.55700000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.374,"15min":0.36299999999999999},"min":{"1min":0.23499999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.48599999999999999}]},{"osd":2,"last update":"Sun Apr 14 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40799999999999997,"15min":0.41199999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.60499999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.43099999999999999,"15min":0.436},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.53600000000000003,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.45200000000000001}]},{"osd":3,"last update":"Sun Apr 14 15:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.46300000000000002,"15min":0.46500000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.59699999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.5}]},{"osd":4,"last update":"Sun Apr 14 15:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61299999999999999,"15min":0.61899999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61499999999999999,"15min":0.61799999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.83599999999999997,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.86899999999999999}]},{"osd":5,"last update":"Sun Apr 14 15:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58899999999999997,"15min":0.59399999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.85699999999999998}]},{"osd":6,"last update":"Sun Apr 14 15:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.75900000000000001}]},{"osd":7,"last update":"Sun Apr 14 15:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.78100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Apr 14 15:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35799999999999998,"15min":0.34699999999999998},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.36899999999999999,"15min":0.35499999999999998},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.51400000000000001,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.27600000000000002}]},{"osd":2,"last update":"Sun Apr 14 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.43099999999999999,"15min":0.437},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.34000000000000002}]},{"osd":3,"last update":"Sun Apr 14 15:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.61299999999999999,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.41999999999999998,"15min":0.41699999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.59099999999999997,"5min":0.629,"15min":0.629},"last":0.309}]},{"osd":4,"last update":"Sun Apr 14 15:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.44900000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.71599999999999997,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.50700000000000001}]},{"osd":5,"last update":"Sun Apr 14 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.45400000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61199999999999999,"15min":0.61799999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.71499999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.44400000000000001}]},{"osd":6,"last update":"Sun Apr 14 15:24:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.59499999999999997}]},{"osd":7,"last update":"Sun Apr 14 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.53400000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.373,"15min":0.36099999999999999},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39500000000000002,"15min":0.38600000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.48499999999999999}]},{"osd":1,"last update":"Sun Apr 14 15:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.27800000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.61499999999999999}]},{"osd":3,"last update":"Sun Apr 14 15:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.56399999999999995}]},{"osd":4,"last update":"Sun Apr 14 15:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.85399999999999998}]},{"osd":5,"last update":"Sun Apr 14 15:25:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65300000000000002,"15min":0.65900000000000003},"min":{"1min":0.42799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76200000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.879},{"interface":"front","average":{"1min":0.622,"5min":0.68999999999999995,"15min":0.70199999999999996},"min":{"1min":0.51900000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77400000000000002,"5min":1.02,"15min":1.02},"last":0.94799999999999995}]},{"osd":6,"last update":"Sun Apr 14 15:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.89900000000000002}]},{"osd":7,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.91100000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.39300000000000002,"15min":0.38800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.70199999999999996,"5min":0.749,"15min":0.749},"last":0.36699999999999999}]},{"osd":1,"last update":"Sun Apr 14 15:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46400000000000002,"15min":0.46300000000000002},"min":{"1min":0.35099999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.67300000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.41799999999999998}]},{"osd":2,"last update":"Sun Apr 14 15:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51200000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72599999999999998,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.40100000000000002}]},{"osd":4,"last update":"Sun Apr 14 15:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":1.4259999999999999,"15min":1.5549999999999999},"min":{"1min":0.45900000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97199999999999998,"5min":19.616,"15min":19.616},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.44400000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.48899999999999999}]},{"osd":5,"last update":"Sun Apr 14 15:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":1.5189999999999999,"15min":1.667},"min":{"1min":0.48799999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82599999999999996,"5min":19.576000000000001,"15min":19.576000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.5129999999999999,"15min":1.6579999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.083,"5min":19.699000000000002,"15min":19.699000000000002},"last":0.63}]},{"osd":6,"last update":"Sun Apr 14 15:24:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.52100000000000002}]},{"osd":7,"last update":"Sun Apr 14 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.56699999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.51000000000000001,"15min":0.503},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.51300000000000001,"15min":0.50900000000000001},"min":{"1min":0.31900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.49299999999999999}]},{"osd":1,"last update":"Sun Apr 14 15:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.36599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.52800000000000002}]},{"osd":2,"last update":"Sun Apr 14 15:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.35199999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69399999999999995,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.41199999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.56200000000000006}]},{"osd":3,"last update":"Sun Apr 14 15:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.5,"15min":0.49099999999999999},"min":{"1min":0.38700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68899999999999995,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.53600000000000003}]},{"osd":5,"last update":"Sun Apr 14 15:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.41899999999999998,"15min":0.40699999999999997},"min":{"1min":0.30099999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.435},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.45400000000000001,"15min":0.443},"min":{"1min":0.35399999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.443}]},{"osd":6,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.377},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.39600000000000002}]},{"osd":7,"last update":"Sun Apr 14 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.41999999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5700,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739452,"statfs":{"total":95995035648,"available":95989198848,"internally_reserved":0,"allocated":585728,"data_stored":240284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72499999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.56299999999999994}]},{"osd":1,"last update":"Sun Apr 14 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.67400000000000004,"15min":0.622},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":9.9749999999999996,"5min":9.9749999999999996,"15min":9.9749999999999996},"last":0.51200000000000001}]},{"osd":2,"last update":"Sun Apr 14 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.63900000000000001,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":9.9369999999999994,"5min":9.9369999999999994,"15min":9.9369999999999994},"last":0.746},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72499999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58099999999999996}]},{"osd":3,"last update":"Sun Apr 14 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.68899999999999995,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.72499999999999998}]},{"osd":4,"last update":"Sun Apr 14 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.41199999999999998,"15min":0.40400000000000003},"min":{"1min":0.32500000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.442,"15min":0.433},"min":{"1min":0.34100000000000003,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70999999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.76700000000000002}]},{"osd":6,"last update":"Sun Apr 14 15:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.755},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.60199999999999998}]},{"osd":7,"last update":"Sun Apr 14 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:25:17.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T15:25:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:17 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:17 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:17 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:17 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:17 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:25:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:17 smithi196 ceph-mon[25696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:25:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:17 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:17 smithi196 ceph-mon[25696]: Deploying daemon grafana.a on smithi196 2024-04-14T15:25:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:17 smithi196 ceph-mon[25696]: from='client.24517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:25:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:17 smithi196 ceph-mon[25696]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:18.014 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:19.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:25:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:25:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:25:19.682 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:25:19.684 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-14T15:25:20.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:19 smithi016 ceph-mon[29517]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:20.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:19 smithi016 ceph-mon[24869]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:20.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:19 smithi196 ceph-mon[25696]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:20.223 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-14T15:25:19.252199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48980,"kb_used_data":6164,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912236,"statfs":{"total":767960285184,"available":767910129664,"internally_reserved":0,"allocated":6311936,"data_stored":3511317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002125"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":263,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-14T15:24:07.323617+0000","last_change":"2024-04-14T15:23:51.916120+0000","last_active":"2024-04-14T15:24:07.323617+0000","last_peered":"2024-04-14T15:24:07.323617+0000","last_clean":"2024-04-14T15:24:07.323617+0000","last_became_active":"2024-04-14T15:23:51.915586+0000","last_became_peered":"2024-04-14T15:23:51.915586+0000","last_unstale":"2024-04-14T15:24:07.323617+0000","last_undegraded":"2024-04-14T15:24:07.323617+0000","last_fullsized":"2024-04-14T15:24:07.323617+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T15:21:54.677147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T15:21:54.677147+0000","last_clean_scrub_stamp":"2024-04-14T15:21:54.677147+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:57:46.478125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.54600000000000004}]},{"osd":1,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.59099999999999997}]},{"osd":2,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.57799999999999996}]},{"osd":3,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.46899999999999997}]},{"osd":4,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.49199999999999999}]},{"osd":5,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.435}]},{"osd":6,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.42399999999999999}]}]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6160,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738992,"statfs":{"total":95995035648,"available":95988727808,"internally_reserved":0,"allocated":1122304,"data_stored":769535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.49099999999999999}]},{"osd":1,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.44}]},{"osd":2,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.55500000000000005}]},{"osd":4,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.40100000000000002}]},{"osd":5,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.5}]},{"osd":7,"last update":"Sun Apr 14 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.54200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.35699999999999998,"15min":0.33700000000000002},"min":{"1min":0.24399999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.55700000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.375,"5min":0.374,"15min":0.36299999999999999},"min":{"1min":0.23499999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Sun Apr 14 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40799999999999997,"15min":0.41199999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.60499999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.43099999999999999,"15min":0.436},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.53600000000000003,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.377}]},{"osd":3,"last update":"Sun Apr 14 15:24:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.46300000000000002,"15min":0.46500000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.59699999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.33200000000000002}]},{"osd":4,"last update":"Sun Apr 14 15:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61299999999999999,"15min":0.61899999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61499999999999999,"15min":0.61799999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.83599999999999997,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.49299999999999999}]},{"osd":5,"last update":"Sun Apr 14 15:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58899999999999997,"15min":0.59399999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.47999999999999998}]},{"osd":6,"last update":"Sun Apr 14 15:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45000000000000001}]},{"osd":7,"last update":"Sun Apr 14 15:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Apr 14 15:24:32 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35799999999999998,"15min":0.34699999999999998},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.251},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.36899999999999999,"15min":0.35499999999999998},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.51400000000000001,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.26700000000000002}]},{"osd":2,"last update":"Sun Apr 14 15:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.43099999999999999,"15min":0.437},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.28599999999999998}]},{"osd":3,"last update":"Sun Apr 14 15:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.61299999999999999,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.438,"5min":0.41999999999999998,"15min":0.41699999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.59099999999999997,"5min":0.629,"15min":0.629},"last":0.318}]},{"osd":4,"last update":"Sun Apr 14 15:24:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.44900000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.495},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.71599999999999997,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.47499999999999998}]},{"osd":5,"last update":"Sun Apr 14 15:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60199999999999998,"15min":0.60599999999999998},"min":{"1min":0.45400000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61199999999999999,"15min":0.61799999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.71499999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.505}]},{"osd":6,"last update":"Sun Apr 14 15:24:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48199999999999998}]},{"osd":7,"last update":"Sun Apr 14 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.432}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.373,"15min":0.36099999999999999},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39500000000000002,"15min":0.38600000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.48499999999999999}]},{"osd":1,"last update":"Sun Apr 14 15:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.27800000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.61499999999999999}]},{"osd":3,"last update":"Sun Apr 14 15:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.56399999999999995}]},{"osd":4,"last update":"Sun Apr 14 15:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60699999999999998,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.85399999999999998}]},{"osd":5,"last update":"Sun Apr 14 15:25:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65300000000000002,"15min":0.65900000000000003},"min":{"1min":0.42799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76200000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.879},{"interface":"front","average":{"1min":0.622,"5min":0.68999999999999995,"15min":0.70199999999999996},"min":{"1min":0.51900000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77400000000000002,"5min":1.02,"15min":1.02},"last":0.94799999999999995}]},{"osd":6,"last update":"Sun Apr 14 15:24:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.89900000000000002}]},{"osd":7,"last update":"Sun Apr 14 15:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.91100000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.39300000000000002,"15min":0.38800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.70199999999999996,"5min":0.749,"15min":0.749},"last":0.58499999999999996}]},{"osd":1,"last update":"Sun Apr 14 15:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46400000000000002,"15min":0.46300000000000002},"min":{"1min":0.35099999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.67300000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.503},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Apr 14 15:24:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51300000000000001,"15min":0.51200000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.50600000000000001,"15min":0.50900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72599999999999998,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.45100000000000001}]},{"osd":4,"last update":"Sun Apr 14 15:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":1.4259999999999999,"15min":1.5549999999999999},"min":{"1min":0.45900000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97199999999999998,"5min":19.616,"15min":19.616},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.57899999999999996,"15min":0.56999999999999995},"min":{"1min":0.44400000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.67000000000000004}]},{"osd":5,"last update":"Sun Apr 14 15:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":1.5189999999999999,"15min":1.667},"min":{"1min":0.48799999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82599999999999996,"5min":19.576000000000001,"15min":19.576000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.5129999999999999,"15min":1.6579999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.083,"5min":19.699000000000002,"15min":19.699000000000002},"last":0.68500000000000005}]},{"osd":6,"last update":"Sun Apr 14 15:24:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.79900000000000004}]},{"osd":7,"last update":"Sun Apr 14 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.749}]}]},{"osd":4,"up_from":31,"seq":133143986211,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.51000000000000001,"15min":0.503},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.51300000000000001,"15min":0.50900000000000001},"min":{"1min":0.31900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Sun Apr 14 15:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.51300000000000001,"15min":0.504},"min":{"1min":0.36599999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.504}]},{"osd":2,"last update":"Sun Apr 14 15:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.35199999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69399999999999995,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.51900000000000002,"15min":0.50700000000000001},"min":{"1min":0.41199999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Apr 14 15:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.5,"15min":0.49099999999999999},"min":{"1min":0.38700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68899999999999995,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.59099999999999997}]},{"osd":5,"last update":"Sun Apr 14 15:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.41899999999999998,"15min":0.40699999999999997},"min":{"1min":0.30099999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.45400000000000001,"15min":0.443},"min":{"1min":0.35399999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.41599999999999998}]},{"osd":6,"last update":"Sun Apr 14 15:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.42999999999999999}]},{"osd":7,"last update":"Sun Apr 14 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45600000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5700,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739452,"statfs":{"total":95995035648,"available":95989198848,"internally_reserved":0,"allocated":585728,"data_stored":240284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72499999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.51600000000000001}]},{"osd":1,"last update":"Sun Apr 14 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.67400000000000004,"15min":0.622},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":9.9749999999999996,"5min":9.9749999999999996,"15min":9.9749999999999996},"last":0.59199999999999997}]},{"osd":2,"last update":"Sun Apr 14 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.63900000000000001,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":9.9369999999999994,"5min":9.9369999999999994,"15min":9.9369999999999994},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72499999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.66100000000000003}]},{"osd":3,"last update":"Sun Apr 14 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.68899999999999995,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.70099999999999996}]},{"osd":4,"last update":"Sun Apr 14 15:25:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.41199999999999998,"15min":0.40400000000000003},"min":{"1min":0.32500000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.442,"15min":0.433},"min":{"1min":0.34100000000000003,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70999999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.629}]},{"osd":6,"last update":"Sun Apr 14 15:24:28 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.72199999999999998}]},{"osd":7,"last update":"Sun Apr 14 15:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T15:25:20.225 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-14T15:25:20.225 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-14T15:25:20.225 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-14T15:25:20.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- ceph health --format=json 2024-04-14T15:25:20.449 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T15:25:20.795 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:20 smithi016 ceph-mon[29517]: from='client.14607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:25:20.796 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:20 smithi016 ceph-mon[24869]: from='client.14607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:25:21.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:20 smithi196 ceph-mon[25696]: from='client.14607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T15:25:21.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:21 smithi016 ceph-mon[24869]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:21.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:21 smithi016 ceph-mon[29517]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:22.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:21 smithi196 ceph-mon[25696]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:22.197 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-14T15:25:22.198 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-14T15:25:22.513 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:25:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:25:22] "GET /metrics HTTP/1.1" 200 191489 "" "Prometheus/2.33.4" 2024-04-14T15:25:22.806 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:22 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1035157046' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-14T15:25:22.806 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:22 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1035157046' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-14T15:25:22.829 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-14T15:25:22.829 INFO:tasks.cephadm:Setup complete, yielding 2024-04-14T15:25:22.830 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T15:25:22.840 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-14T15:25:22.840 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-04-14T15:25:23.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:22 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1035157046' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-14T15:25:24.012 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:23 smithi016 ceph-mon[29517]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:24.013 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:23 smithi016 ceph-mon[24869]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:24.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:23 smithi196 ceph-mon[25696]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:25.169 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T15:25:25.178 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-14T15:25:25.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-04-14T15:25:25.757 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:25 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1054863152' entity='client.admin' 2024-04-14T15:25:25.757 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:25 smithi016 ceph-mon[24869]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:25.757 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:25 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1054863152' entity='client.admin' 2024-04-14T15:25:25.757 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:25 smithi016 ceph-mon[29517]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:25.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:25 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1054863152' entity='client.admin' 2024-04-14T15:25:25.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:25 smithi196 ceph-mon[25696]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:28.197 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:25:28.197 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "05a55308-7343-4eb3-a110-42efb7bb9b17", 2024-04-14T15:25:28.197 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "r", 2024-04-14T15:25:28.197 INFO:teuthology.orchestra.run.smithi016.stdout: "current_period": "862f7c74-6be4-4870-8616-3e1f3fe43063", 2024-04-14T15:25:28.197 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 1 2024-04-14T15:25:28.197 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:25:28.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:28 smithi196 ceph-mon[25696]: osdmap e61: 8 total, 8 up, 8 in 2024-04-14T15:25:28.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:28 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/370486396' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T15:25:28.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:28 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T15:25:28.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:28 smithi196 ceph-mon[25696]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:28.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:28 smithi016 ceph-mon[29517]: osdmap e61: 8 total, 8 up, 8 in 2024-04-14T15:25:28.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:28 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/370486396' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T15:25:28.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:28 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T15:25:28.533 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:28 smithi016 ceph-mon[29517]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:28.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:28 smithi016 ceph-mon[24869]: osdmap e61: 8 total, 8 up, 8 in 2024-04-14T15:25:28.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:28 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/370486396' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T15:25:28.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:28 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T15:25:28.533 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:28 smithi016 ceph-mon[24869]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:28.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-04-14T15:25:29.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:29 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-14T15:25:29.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:29 smithi196 ceph-mon[25696]: osdmap e62: 8 total, 8 up, 8 in 2024-04-14T15:25:29.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:25:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:25:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:25:29.495 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:29 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-14T15:25:29.495 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:29 smithi016 ceph-mon[29517]: osdmap e62: 8 total, 8 up, 8 in 2024-04-14T15:25:29.495 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:29 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-14T15:25:29.495 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:29 smithi016 ceph-mon[24869]: osdmap e62: 8 total, 8 up, 8 in 2024-04-14T15:25:30.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:30 smithi196 ceph-mon[25696]: osdmap e63: 8 total, 8 up, 8 in 2024-04-14T15:25:30.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:30 smithi196 ceph-mon[25696]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T15:25:30.435 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:30 smithi016 ceph-mon[29517]: osdmap e63: 8 total, 8 up, 8 in 2024-04-14T15:25:30.435 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:30 smithi016 ceph-mon[29517]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T15:25:30.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:30 smithi016 ceph-mon[24869]: osdmap e63: 8 total, 8 up, 8 in 2024-04-14T15:25:30.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:30 smithi016 ceph-mon[24869]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T15:25:30.518 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:25:30.518 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "eabbda11-2cbe-43b5-8fb6-a4dc3fe1447e", 2024-04-14T15:25:30.518 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default", 2024-04-14T15:25:30.518 INFO:teuthology.orchestra.run.smithi016.stdout: "api_name": "default", 2024-04-14T15:25:30.518 INFO:teuthology.orchestra.run.smithi016.stdout: "is_master": "true", 2024-04-14T15:25:30.518 INFO:teuthology.orchestra.run.smithi016.stdout: "endpoints": [], 2024-04-14T15:25:30.519 INFO:teuthology.orchestra.run.smithi016.stdout: "hostnames": [], 2024-04-14T15:25:30.519 INFO:teuthology.orchestra.run.smithi016.stdout: "hostnames_s3website": [], 2024-04-14T15:25:30.519 INFO:teuthology.orchestra.run.smithi016.stdout: "master_zone": "", 2024-04-14T15:25:30.519 INFO:teuthology.orchestra.run.smithi016.stdout: "zones": [], 2024-04-14T15:25:30.519 INFO:teuthology.orchestra.run.smithi016.stdout: "placement_targets": [], 2024-04-14T15:25:30.519 INFO:teuthology.orchestra.run.smithi016.stdout: "default_placement": "", 2024-04-14T15:25:30.519 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_id": "05a55308-7343-4eb3-a110-42efb7bb9b17", 2024-04-14T15:25:30.519 INFO:teuthology.orchestra.run.smithi016.stdout: "sync_policy": { 2024-04-14T15:25:30.519 INFO:teuthology.orchestra.run.smithi016.stdout: "groups": [] 2024-04-14T15:25:30.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:25:30.519 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:25:31.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-04-14T15:25:31.557 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:25:31.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 podman[41825]: 2024-04-14 15:25:31.459182644 +0000 UTC m=+0.016364470 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:25:31.824 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 podman[41825]: 2024-04-14 15:25:31.558109927 +0000 UTC m=+0.115291752 container create 6cd5c62d1ec5d873d24036adf57e1eaa91a2a88c296c6430b3e8cbdcb3cb1db0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, com.redhat.component=ubi8-container, description=Ceph Grafana Container, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, vcs-type=git, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, io.openshift.expose-services=, io.buildah.version=1.24.2) 2024-04-14T15:25:31.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 podman[41825]: 2024-04-14 15:25:31.793814042 +0000 UTC m=+0.350995867 container init 6cd5c62d1ec5d873d24036adf57e1eaa91a2a88c296c6430b3e8cbdcb3cb1db0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.buildah.version=1.24.2, release=236.1648460182, vcs-type=git, maintainer=Paul Cuzner , vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, version=8.5, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container) 2024-04-14T15:25:31.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 podman[41825]: 2024-04-14 15:25:31.796512314 +0000 UTC m=+0.353694150 container start 6cd5c62d1ec5d873d24036adf57e1eaa91a2a88c296c6430b3e8cbdcb3cb1db0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, maintainer=Paul Cuzner , name=ubi8, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, io.openshift.expose-services=, vcs-type=git, com.redhat.component=ubi8-container) 2024-04-14T15:25:31.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-14T15:25:31.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-14T15:25:31.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-14T15:25:31.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-14T15:25:31.825 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-14T15:25:32.074 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="App mode production" logger=settings 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-04-14T15:25:32.075 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-04-14T15:25:32.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-04-14T15:25:32.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-04-14T15:25:32.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-04-14T15:25:32.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-04-14T15:25:32.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-04-14T15:25:32.076 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-04-14T15:25:32.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-04-14T15:25:32.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-04-14T15:25:32.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-14T15:25:32.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-04-14T15:25:32.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-04-14T15:25:32.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-04-14T15:25:32.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-04-14T15:25:32.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-04-14T15:25:32.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-04-14T15:25:32.077 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-14T15:25:32.078 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-04-14T15:25:32.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-04-14T15:25:32.080 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 bash[41825]: 6cd5c62d1ec5d873d24036adf57e1eaa91a2a88c296c6430b3e8cbdcb3cb1db0 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-04-14T15:25:32.081 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-04-14T15:25:32.082 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:31+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-04-14T15:25:32.083 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-04-14T15:25:32.084 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-04-14T15:25:32.084 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-04-14T15:25:32.084 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-04-14T15:25:32.084 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-04-14T15:25:32.084 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-04-14T15:25:32.084 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-04-14T15:25:32.084 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-04-14T15:25:32.084 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-04-14T15:25:32.085 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-04-14T15:25:32.086 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-04-14T15:25:32.086 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-04-14T15:25:32.086 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-04-14T15:25:32.086 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-04-14T15:25:32.086 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-04-14T15:25:32.324 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-mon[25696]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T15:25:32.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:32.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:32.325 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:32.325 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-14T15:25:32.326 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-04-14T15:25:32.327 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-04-14T15:25:32.328 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-14T15:25:32.329 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-14T15:25:32.329 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-14T15:25:32.330 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-04-14T15:25:32.330 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-14T15:25:32.330 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-04-14T15:25:32.330 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-14T15:25:32.331 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-04-14T15:25:32.332 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-04-14T15:25:32.333 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-04-14T15:25:32.334 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-04-14T15:25:32.335 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-04-14T15:25:32.336 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-04-14T15:25:32.337 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-04-14T15:25:32.338 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-14T15:25:32.339 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-04-14T15:25:32.340 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-14T15:25:32.341 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-04-14T15:25:32.342 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-14T15:25:32.342 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-14T15:25:32.342 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-14T15:25:32.342 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-04-14T15:25:32.342 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-04-14T15:25:32.342 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-04-14T15:25:32.342 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-04-14T15:25:32.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:25:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:25:32] "GET /metrics HTTP/1.1" 200 191489 "" "Prometheus/2.33.4" 2024-04-14T15:25:32.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:32 smithi016 ceph-mon[24869]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T15:25:32.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:32 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:32.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:32 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:32.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:32 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:32.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:32 smithi016 ceph-mon[29517]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T15:25:32.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:32 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:32.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:32 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:32.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:32 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:32.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-04-14T15:25:32.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-14T15:25:32.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-04-14T15:25:32.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-04-14T15:25:32.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-04-14T15:25:32.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-04-14T15:25:32.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-04-14T15:25:32.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-04-14T15:25:32.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-04-14T15:25:32.666 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-04-14T15:25:32.666 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-04-14T15:25:32.666 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-04-14T15:25:32.666 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-04-14T15:25:32.666 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-04-14T15:25:32.666 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-04-14T15:25:32.666 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-04-14T15:25:32.666 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-04-14T15:25:32.666 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-04-14T15:25:32.666 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-04-14T15:25:32.667 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-04-14T15:25:32.667 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-04-14T15:25:32.667 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-04-14T15:25:32.667 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-04-14T15:25:32.667 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-04-14T15:25:32.667 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-04-14T15:25:32.667 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-04-14T15:25:32.667 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-04-14T15:25:32.667 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-04-14T15:25:32.668 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=582.619609ms 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-14T15:25:32.669 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-14T15:25:32.670 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:25:32 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:25:32+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-14T15:25:32.784 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:25:32.784 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "226df31a-2a1a-4ca1-a3a5-7cf591d71130", 2024-04-14T15:25:32.784 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "z", 2024-04-14T15:25:32.784 INFO:teuthology.orchestra.run.smithi016.stdout: "domain_root": "z.rgw.meta:root", 2024-04-14T15:25:32.784 INFO:teuthology.orchestra.run.smithi016.stdout: "control_pool": "z.rgw.control", 2024-04-14T15:25:32.784 INFO:teuthology.orchestra.run.smithi016.stdout: "gc_pool": "z.rgw.log:gc", 2024-04-14T15:25:32.784 INFO:teuthology.orchestra.run.smithi016.stdout: "lc_pool": "z.rgw.log:lc", 2024-04-14T15:25:32.784 INFO:teuthology.orchestra.run.smithi016.stdout: "log_pool": "z.rgw.log", 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "roles_pool": "z.rgw.meta:roles", 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "otp_pool": "z.rgw.otp", 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "system_key": { 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "access_key": "", 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: "secret_key": "" 2024-04-14T15:25:32.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: "placement_pools": [ 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "default-placement", 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: "val": { 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: "index_pool": "z.rgw.buckets.index", 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_classes": { 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: "STANDARD": { 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: "data_pool": "z.rgw.buckets.data" 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: "index_type": 0 2024-04-14T15:25:32.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:25:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:25:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:25:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_id": "05a55308-7343-4eb3-a110-42efb7bb9b17", 2024-04-14T15:25:32.787 INFO:teuthology.orchestra.run.smithi016.stdout: "notif_pool": "z.rgw.log:notif" 2024-04-14T15:25:32.787 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:25:33.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:33 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:33.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-04-14T15:25:33.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:33 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:33.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:33 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:34.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:34 smithi016 ceph-mon[29517]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-04-14T15:25:34.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:34 smithi016 ceph-mon[24869]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-04-14T15:25:34.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:34 smithi196 ceph-mon[25696]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-04-14T15:25:36.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:36 smithi016 ceph-mon[24869]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-14T15:25:36.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:36 smithi016 ceph-mon[29517]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-14T15:25:36.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:36 smithi196 ceph-mon[25696]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-14T15:25:37.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:37 smithi016 ceph-mon[24869]: osdmap e64: 8 total, 8 up, 8 in 2024-04-14T15:25:37.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:37 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1190047203' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:25:37.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:37 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:25:37.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:37 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:37.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:37 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:37.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:37 smithi016 ceph-mon[29517]: osdmap e64: 8 total, 8 up, 8 in 2024-04-14T15:25:37.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:37 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1190047203' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:25:37.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:37 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:25:37.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:37 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:37.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:37 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:37.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:37 smithi196 ceph-mon[25696]: osdmap e64: 8 total, 8 up, 8 in 2024-04-14T15:25:37.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:37 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1190047203' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:25:37.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:37 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-14T15:25:37.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:37 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:37.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:37 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:38.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:38 smithi016 ceph-mon[24869]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 1012 B/s wr, 2 op/s 2024-04-14T15:25:38.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:38 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-14T15:25:38.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:38 smithi016 ceph-mon[24869]: osdmap e65: 8 total, 8 up, 8 in 2024-04-14T15:25:38.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:38 smithi016 ceph-mon[29517]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 1012 B/s wr, 2 op/s 2024-04-14T15:25:38.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:38 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-14T15:25:38.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:38 smithi016 ceph-mon[29517]: osdmap e65: 8 total, 8 up, 8 in 2024-04-14T15:25:38.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:38 smithi196 ceph-mon[25696]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 1012 B/s wr, 2 op/s 2024-04-14T15:25:38.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:38 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-14T15:25:38.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:38 smithi196 ceph-mon[25696]: osdmap e65: 8 total, 8 up, 8 in 2024-04-14T15:25:39.342 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:25:39 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:25:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:25:39.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:39 smithi196 ceph-mon[25696]: osdmap e66: 8 total, 8 up, 8 in 2024-04-14T15:25:39.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:39 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1190047203' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:25:39.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:39 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:25:39.344 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:39 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:39.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:39 smithi016 ceph-mon[24869]: osdmap e66: 8 total, 8 up, 8 in 2024-04-14T15:25:39.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:39 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1190047203' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:25:39.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:39 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:25:39.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:39 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:39.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:39 smithi016 ceph-mon[29517]: osdmap e66: 8 total, 8 up, 8 in 2024-04-14T15:25:39.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:39 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1190047203' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:25:39.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:39 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-14T15:25:39.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:39 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:40.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:40 smithi016 ceph-mon[24869]: pgmap v55: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-04-14T15:25:40.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:40 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-14T15:25:40.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:40 smithi016 ceph-mon[24869]: osdmap e67: 8 total, 8 up, 8 in 2024-04-14T15:25:40.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:40 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:40.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:40 smithi016 ceph-mon[29517]: pgmap v55: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-04-14T15:25:40.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:40 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-14T15:25:40.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:40 smithi016 ceph-mon[29517]: osdmap e67: 8 total, 8 up, 8 in 2024-04-14T15:25:40.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:40 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:40 smithi196 ceph-mon[25696]: pgmap v55: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-04-14T15:25:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:40 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-14T15:25:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:40 smithi196 ceph-mon[25696]: osdmap e67: 8 total, 8 up, 8 in 2024-04-14T15:25:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:40 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:41.422 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[29517]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T15:25:41.422 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[29517]: Reconfiguring daemon alertmanager.a on smithi016 2024-04-14T15:25:41.422 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[29517]: osdmap e68: 8 total, 8 up, 8 in 2024-04-14T15:25:41.422 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1658881450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:25:41.422 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:25:41.422 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-14T15:25:41.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[24869]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T15:25:41.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[24869]: Reconfiguring daemon alertmanager.a on smithi016 2024-04-14T15:25:41.423 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[24869]: osdmap e68: 8 total, 8 up, 8 in 2024-04-14T15:25:41.423 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1658881450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:25:41.423 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:25:41.423 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:41 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-14T15:25:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:41 smithi196 ceph-mon[25696]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T15:25:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:41 smithi196 ceph-mon[25696]: Reconfiguring daemon alertmanager.a on smithi016 2024-04-14T15:25:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:41 smithi196 ceph-mon[25696]: osdmap e68: 8 total, 8 up, 8 in 2024-04-14T15:25:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:41 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1658881450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:25:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:41 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T15:25:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:41 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-14T15:25:42.583 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:25:42] "GET /metrics HTTP/1.1" 200 194180 "" "Prometheus/2.33.4" 2024-04-14T15:25:42.583 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[29517]: pgmap v58: 129 pgs: 64 unknown, 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-04-14T15:25:42.583 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[29517]: osdmap e69: 8 total, 8 up, 8 in 2024-04-14T15:25:42.584 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1658881450' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:25:42.584 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:25:42.584 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T15:25:42.584 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[29517]: osdmap e70: 8 total, 8 up, 8 in 2024-04-14T15:25:42.584 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1658881450' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T15:25:42.584 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T15:25:42.584 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:42 smithi016 systemd[1]: Stopping Ceph alertmanager.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:25:42.584 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:42 smithi016 bash[49559]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager.a" found: no such container 2024-04-14T15:25:42.584 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[24869]: pgmap v58: 129 pgs: 64 unknown, 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-04-14T15:25:42.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[24869]: osdmap e69: 8 total, 8 up, 8 in 2024-04-14T15:25:42.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1658881450' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:25:42.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:25:42.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T15:25:42.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[24869]: osdmap e70: 8 total, 8 up, 8 in 2024-04-14T15:25:42.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1658881450' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T15:25:42.585 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T15:25:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:42 smithi196 ceph-mon[25696]: pgmap v58: 129 pgs: 64 unknown, 65 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-04-14T15:25:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:42 smithi196 ceph-mon[25696]: osdmap e69: 8 total, 8 up, 8 in 2024-04-14T15:25:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:42 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1658881450' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:25:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:42 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T15:25:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:42 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T15:25:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:42 smithi196 ceph-mon[25696]: osdmap e70: 8 total, 8 up, 8 in 2024-04-14T15:25:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:42 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1658881450' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T15:25:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:42 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T15:25:42.834 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[46954]: level=info ts=2024-04-14T15:25:42.583Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-14T15:25:42.834 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:42 smithi016 podman[49573]: 2024-04-14 15:25:42.614191349 +0000 UTC m=+0.130901732 container died dd21ba7bfc250dcfc17883937b90bf924a717bda10f1c79d17a5e340ab27920b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:43.110 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:42 smithi016 podman[49573]: 2024-04-14 15:25:42.907739645 +0000 UTC m=+0.424450022 container cleanup dd21ba7bfc250dcfc17883937b90bf924a717bda10f1c79d17a5e340ab27920b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:43.111 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:42 smithi016 bash[49573]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a 2024-04-14T15:25:43.111 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:42 smithi016 bash[49599]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager.a" found: no such container 2024-04-14T15:25:43.412 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 podman[49586]: 2024-04-14 15:25:43.150910525 +0000 UTC m=+0.539431030 container remove dd21ba7bfc250dcfc17883937b90bf924a717bda10f1c79d17a5e340ab27920b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:43.412 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 podman[49586]: 2024-04-14 15:25:43.234715668 +0000 UTC m=+0.623236185 volume remove dce330fab22f6df9aa5f3bee7d8544a2fea47dc741c25c6a5b070ce81d428288 2024-04-14T15:25:43.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-14T15:25:43.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 systemd[1]: Stopped Ceph alertmanager.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:25:43.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 systemd[1]: Starting Ceph alertmanager.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:25:43.413 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:25:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "fc14b12b-85e3-442c-8ac7-9526a4d11bd7", 2024-04-14T15:25:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 1, 2024-04-14T15:25:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "predecessor_uuid": "862f7c74-6be4-4870-8616-3e1f3fe43063", 2024-04-14T15:25:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "sync_status": [], 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "period_map": { 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "fc14b12b-85e3-442c-8ac7-9526a4d11bd7", 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "zonegroups": [ 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "eabbda11-2cbe-43b5-8fb6-a4dc3fe1447e", 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default", 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "api_name": "default", 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "is_master": "true", 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "endpoints": [], 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "hostnames": [], 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "hostnames_s3website": [], 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "master_zone": "226df31a-2a1a-4ca1-a3a5-7cf591d71130", 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "zones": [ 2024-04-14T15:25:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "id": "226df31a-2a1a-4ca1-a3a5-7cf591d71130", 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "z", 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "endpoints": [], 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "log_meta": "false", 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "log_data": "false", 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "bucket_index_max_shards": 11, 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "read_only": "false", 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_type": "", 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "sync_from_all": "true", 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "sync_from": [], 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "redirect_zone": "" 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "placement_targets": [ 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "default-placement", 2024-04-14T15:25:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "tags": [], 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "storage_classes": [ 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "STANDARD" 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "default_placement": "default-placement", 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_id": "05a55308-7343-4eb3-a110-42efb7bb9b17", 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "sync_policy": { 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "groups": [] 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "short_zone_ids": [ 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "226df31a-2a1a-4ca1-a3a5-7cf591d71130", 2024-04-14T15:25:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "val": 1604003950 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "master_zonegroup": "eabbda11-2cbe-43b5-8fb6-a4dc3fe1447e", 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "master_zone": "226df31a-2a1a-4ca1-a3a5-7cf591d71130", 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "period_config": { 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "bucket_quota": { 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": false, 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "check_on_raw": false, 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max_size": -1, 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max_size_kb": 0, 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max_objects": -1 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "user_quota": { 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": false, 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "check_on_raw": false, 2024-04-14T15:25:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "max_size": -1, 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max_size_kb": 0, 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max_objects": -1 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "user_ratelimit": { 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_ops": 0, 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_ops": 0, 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_bytes": 0, 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_bytes": 0, 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": false 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "bucket_ratelimit": { 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_ops": 0, 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_ops": 0, 2024-04-14T15:25:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_bytes": 0, 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_bytes": 0, 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": false 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "anonymous_ratelimit": { 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_ops": 0, 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_ops": 0, 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max_read_bytes": 0, 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "max_write_bytes": 0, 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "enabled": false 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_id": "05a55308-7343-4eb3-a110-42efb7bb9b17", 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_name": "r", 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "realm_epoch": 2 2024-04-14T15:25:43.419 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:25:43.804 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 podman[49670]: 2024-04-14 15:25:43.379053879 +0000 UTC m=+0.022190082 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-14T15:25:43.804 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 podman[49670]: 2024-04-14 15:25:43.502487675 +0000 UTC m=+0.145623861 volume create acd29d35ff693085daa0538dce60b8849e3790484bcc627d5184ab31648b4179 2024-04-14T15:25:43.804 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 podman[49670]: 2024-04-14 15:25:43.586384395 +0000 UTC m=+0.229520583 container create e0503939608c0a3ebbbb1f5c36f412351974b5fcd1e237f84f23ff6ba90092b6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:44.355 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 podman[49670]: 2024-04-14 15:25:43.905969341 +0000 UTC m=+0.549105528 container init e0503939608c0a3ebbbb1f5c36f412351974b5fcd1e237f84f23ff6ba90092b6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:44.356 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 podman[49670]: 2024-04-14 15:25:43.90879464 +0000 UTC m=+0.551930832 container start e0503939608c0a3ebbbb1f5c36f412351974b5fcd1e237f84f23ff6ba90092b6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:44.356 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=info ts=2024-04-14T15:25:43.920Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-14T15:25:44.356 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=info ts=2024-04-14T15:25:43.920Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-14T15:25:44.356 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=info ts=2024-04-14T15:25:43.921Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.16 port=9094 2024-04-14T15:25:44.356 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=info ts=2024-04-14T15:25:43.922Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-14T15:25:44.356 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=info ts=2024-04-14T15:25:43.949Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T15:25:44.356 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=info ts=2024-04-14T15:25:43.949Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T15:25:44.356 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=info ts=2024-04-14T15:25:43.951Z caller=main.go:518 msg=Listening address=:9093 2024-04-14T15:25:44.356 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=info ts=2024-04-14T15:25:43.951Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-14T15:25:44.356 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:43 smithi016 bash[49670]: e0503939608c0a3ebbbb1f5c36f412351974b5fcd1e237f84f23ff6ba90092b6 2024-04-14T15:25:44.356 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:44 smithi016 systemd[1]: Started Ceph alertmanager.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:25:44.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:44 smithi016 ceph-mon[29517]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:25:44.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:44 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-14T15:25:44.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:44 smithi016 ceph-mon[29517]: osdmap e71: 8 total, 8 up, 8 in 2024-04-14T15:25:44.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:44 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:44.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:44 smithi016 ceph-mon[24869]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:25:44.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:44 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-14T15:25:44.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:44 smithi016 ceph-mon[24869]: osdmap e71: 8 total, 8 up, 8 in 2024-04-14T15:25:44.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:44 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:44.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:44 smithi196 ceph-mon[25696]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T15:25:44.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:44 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-14T15:25:44.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:44 smithi196 ceph-mon[25696]: osdmap e71: 8 total, 8 up, 8 in 2024-04-14T15:25:44.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:44 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:44.728 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-04-14T15:25:45.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:45 smithi196 ceph-mon[25696]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:25:45.554 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:45 smithi196 ceph-mon[25696]: Reconfiguring daemon prometheus.a on smithi196 2024-04-14T15:25:45.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:45 smithi016 ceph-mon[24869]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:25:45.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:45 smithi016 ceph-mon[24869]: Reconfiguring daemon prometheus.a on smithi196 2024-04-14T15:25:45.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:45 smithi016 ceph-mon[29517]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:25:45.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:45 smithi016 ceph-mon[29517]: Reconfiguring daemon prometheus.a on smithi196 2024-04-14T15:25:46.034 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:45 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=info ts=2024-04-14T15:25:45.924Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001014349s 2024-04-14T15:25:46.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:46 smithi016 ceph-mon[24869]: pgmap v63: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 208 B/s rd, 416 B/s wr, 1 op/s 2024-04-14T15:25:46.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:46 smithi016 ceph-mon[29517]: pgmap v63: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 208 B/s rd, 416 B/s wr, 1 op/s 2024-04-14T15:25:46.628 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:46 smithi196 ceph-mon[25696]: pgmap v63: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 208 B/s rd, 416 B/s wr, 1 op/s 2024-04-14T15:25:46.683 INFO:teuthology.orchestra.run.smithi016.stdout:Scheduled rgw.foo update... 2024-04-14T15:25:47.197 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 systemd[1]: Stopping Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:25:47.198 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 bash[42506]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T15:25:47.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch apply rgw smpl' 2024-04-14T15:25:47.482 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.197Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T15:25:47.482 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.197Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-14T15:25:47.482 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.197Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-14T15:25:47.483 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.197Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-14T15:25:47.483 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.197Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-14T15:25:47.483 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.197Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-14T15:25:47.483 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.198Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T15:25:47.483 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.198Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-14T15:25:47.483 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.198Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T15:25:47.483 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.198Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T15:25:47.483 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.198Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-14T15:25:47.483 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[40849]: ts=2024-04-14T15:25:47.198Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-14T15:25:47.483 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 podman[42519]: 2024-04-14 15:25:47.207898469 +0000 UTC m=+0.121373589 container died 7d73a9ee43c2af939ada1adb8e3ca15cca69532df02b9c762a984d93bbb8ada2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:47.832 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-mon[25696]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:25:47.833 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-mon[25696]: Saving service rgw.foo spec with placement count:2 2024-04-14T15:25:47.833 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:47.833 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:47 smithi196 ceph-mon[25696]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-14T15:25:47.833 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 podman[42519]: 2024-04-14 15:25:47.590197362 +0000 UTC m=+0.503672487 container cleanup 7d73a9ee43c2af939ada1adb8e3ca15cca69532df02b9c762a984d93bbb8ada2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:47.833 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 bash[42519]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T15:25:47.833 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 bash[42546]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T15:25:47.961 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:47 smithi016 ceph-mon[29517]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:25:47.962 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:47 smithi016 ceph-mon[29517]: Saving service rgw.foo spec with placement count:2 2024-04-14T15:25:47.962 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:47 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:47.962 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:47 smithi016 ceph-mon[29517]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-14T15:25:47.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:47 smithi016 ceph-mon[24869]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:25:47.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:47 smithi016 ceph-mon[24869]: Saving service rgw.foo spec with placement count:2 2024-04-14T15:25:47.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:47 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:47.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:47 smithi016 ceph-mon[24869]: pgmap v64: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-14T15:25:48.091 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 podman[42534]: 2024-04-14 15:25:47.833025497 +0000 UTC m=+0.627512090 container remove 7d73a9ee43c2af939ada1adb8e3ca15cca69532df02b9c762a984d93bbb8ada2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:48.091 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-14T15:25:48.091 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 systemd[1]: Stopped Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:25:48.091 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:47 smithi196 systemd[1]: Starting Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:25:48.092 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 podman[42619]: 2024-04-14 15:25:47.970233194 +0000 UTC m=+0.018868130 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-14T15:25:48.092 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 podman[42619]: 2024-04-14 15:25:48.092143841 +0000 UTC m=+0.140778823 container create 371540bc5a5a30e30777ad3e62399997be823ba79ee232ff1d457200bb3fbd9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:48.345 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 podman[42619]: 2024-04-14 15:25:48.319458874 +0000 UTC m=+0.368093810 container init 371540bc5a5a30e30777ad3e62399997be823ba79ee232ff1d457200bb3fbd9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:48.345 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 podman[42619]: 2024-04-14 15:25:48.322271642 +0000 UTC m=+0.370906579 container start 371540bc5a5a30e30777ad3e62399997be823ba79ee232ff1d457200bb3fbd9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:25:48.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.346Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-14T15:25:48.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.346Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.346Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.346Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi196 (none))" 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.346Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.346Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.347Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.348Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.348Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.351Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.351Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.296µs 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.351Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 bash[42619]: 371540bc5a5a30e30777ad3e62399997be823ba79ee232ff1d457200bb3fbd9f 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 systemd[1]: Started Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.542Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-14T15:25:48.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.542Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-14T15:25:48.667 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.542Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=46.098µs wal_replay_duration=191.167993ms total_replay_duration=191.233374ms 2024-04-14T15:25:48.667 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.544Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T15:25:48.667 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.544Z caller=main.go:947 level=info msg="TSDB started" 2024-04-14T15:25:48.667 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.544Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T15:25:48.667 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.560Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.915971ms db_storage=1.085µs remote_storage=1.238µs web_handler=334ns query_engine=574ns scrape=225.666µs scrape_sd=24.12µs notify=19.888µs notify_sd=8.004µs rules=15.411032ms 2024-04-14T15:25:48.667 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:25:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:25:48.560Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-14T15:25:49.297 INFO:teuthology.orchestra.run.smithi016.stdout:Scheduled rgw.smpl update... 2024-04-14T15:25:49.717 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.717 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:25:49.717 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:25:49.717 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.16:9093"}]: dispatch 2024-04-14T15:25:49.717 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.16:9093"}]: dispatch 2024-04-14T15:25:49.717 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.717 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:25:49.717 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.196:9095"}]: dispatch 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.196:9095"}]: dispatch 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.196:3000"}]: dispatch 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.196:3000"}]: dispatch 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:25:49.718 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: Saving service rgw.smpl spec with placement count:2 2024-04-14T15:25:49.719 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:25:49.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:25:49.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.16:9093"}]: dispatch 2024-04-14T15:25:49.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.16:9093"}]: dispatch 2024-04-14T15:25:49.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:25:49.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:25:49.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.196:9095"}]: dispatch 2024-04-14T15:25:49.719 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.196:9095"}]: dispatch 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.196:3000"}]: dispatch 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.196:3000"}]: dispatch 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: Saving service rgw.smpl spec with placement count:2 2024-04-14T15:25:49.720 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:25:49.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:25:49.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.16:9093"}]: dispatch 2024-04-14T15:25:49.891 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.16:9093"}]: dispatch 2024-04-14T15:25:49.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:25:49.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:25:49.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.196:9095"}]: dispatch 2024-04-14T15:25:49.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.196:9095"}]: dispatch 2024-04-14T15:25:49.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.892 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:25:49.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:25:49.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.196:3000"}]: dispatch 2024-04-14T15:25:49.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.196:3000"}]: dispatch 2024-04-14T15:25:49.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:49.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:49.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:25:49.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-14T15:25:49.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:25:49.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: Saving service rgw.smpl spec with placement count:2 2024-04-14T15:25:49.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:50.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph osd pool create foo' 2024-04-14T15:25:52.550 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:52 smithi016 ceph-mon[29517]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 919 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-04-14T15:25:52.550 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:52 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:52.550 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:52 smithi016 ceph-mon[24869]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 919 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-04-14T15:25:52.551 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:52 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:52.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:52 smithi196 ceph-mon[25696]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 919 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-04-14T15:25:52.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:52 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'foo' created 2024-04-14T15:25:53.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:53 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1448531598' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-14T15:25:53.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:53 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-14T15:25:53.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:53 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1448531598' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-14T15:25:53.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:53 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-14T15:25:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:53 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1448531598' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-14T15:25:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:53 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-14T15:25:54.042 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:25:53 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=info ts=2024-04-14T15:25:53.925Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002426243s 2024-04-14T15:25:54.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'rbd pool init foo' 2024-04-14T15:25:54.326 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-14T15:25:54.326 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-14T15:25:54.326 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T15:25:54.326 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.326 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.326 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.327 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.327 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: Saving service rgw.foo spec with placement count:2 2024-04-14T15:25:54.327 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.327 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:25:54.327 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:25:54.327 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.327 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:54.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-14T15:25:54.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-14T15:25:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T15:25:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: Saving service rgw.foo spec with placement count:2 2024-04-14T15:25:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:25:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:25:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:54 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-14T15:25:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-14T15:25:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T15:25:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: Saving service rgw.foo spec with placement count:2 2024-04-14T15:25:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:25:54.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:25:54.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:54.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:54 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:55.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:55 smithi016 ceph-mon[29517]: Deploying daemon rgw.foo.smithi016.qozpor on smithi016 2024-04-14T15:25:55.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:55 smithi016 ceph-mon[29517]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T15:25:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:55 smithi016 ceph-mon[24869]: Deploying daemon rgw.foo.smithi016.qozpor on smithi016 2024-04-14T15:25:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:55 smithi016 ceph-mon[24869]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T15:25:55.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:55 smithi196 ceph-mon[25696]: Deploying daemon rgw.foo.smithi016.qozpor on smithi016 2024-04-14T15:25:55.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:55 smithi196 ceph-mon[25696]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T15:25:56.499 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:56 smithi016 ceph-mon[29517]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-04-14T15:25:56.499 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:56 smithi016 ceph-mon[29517]: osdmap e74: 8 total, 8 up, 8 in 2024-04-14T15:25:56.499 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:56 smithi016 ceph-mon[24869]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-04-14T15:25:56.499 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:56 smithi016 ceph-mon[24869]: osdmap e74: 8 total, 8 up, 8 in 2024-04-14T15:25:56.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:56 smithi196 ceph-mon[25696]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-04-14T15:25:56.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:56 smithi196 ceph-mon[25696]: osdmap e74: 8 total, 8 up, 8 in 2024-04-14T15:25:57.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:57 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1041668697' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-14T15:25:57.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:57 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-14T15:25:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:57 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1041668697' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-14T15:25:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:57 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-14T15:25:57.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:57 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1041668697' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-14T15:25:57.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:57 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-14T15:25:58.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:58 smithi196 ceph-mon[25696]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:58.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:58 smithi196 ceph-mon[25696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-14T15:25:58.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:58 smithi196 ceph-mon[25696]: osdmap e75: 8 total, 8 up, 8 in 2024-04-14T15:25:58.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:58 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:58.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:58 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:25:58.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:58 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:25:58.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:58 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:58.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:58 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:58.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[24869]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[24869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-14T15:25:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[24869]: osdmap e75: 8 total, 8 up, 8 in 2024-04-14T15:25:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:25:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:25:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[29517]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:25:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[29517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-14T15:25:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[29517]: osdmap e75: 8 total, 8 up, 8 in 2024-04-14T15:25:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:58.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:25:58.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:25:58.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:25:58.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:58 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:25:59.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:59 smithi196 ceph-mon[25696]: Deploying daemon rgw.foo.smithi196.wjbrhf on smithi196 2024-04-14T15:25:59.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:25:59 smithi196 ceph-mon[25696]: osdmap e76: 8 total, 8 up, 8 in 2024-04-14T15:25:59.566 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:25:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:25:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:25:59.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:59 smithi016 ceph-mon[24869]: Deploying daemon rgw.foo.smithi196.wjbrhf on smithi196 2024-04-14T15:25:59.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:25:59 smithi016 ceph-mon[24869]: osdmap e76: 8 total, 8 up, 8 in 2024-04-14T15:25:59.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:59 smithi016 ceph-mon[29517]: Deploying daemon rgw.foo.smithi196.wjbrhf on smithi196 2024-04-14T15:25:59.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:25:59 smithi016 ceph-mon[29517]: osdmap e76: 8 total, 8 up, 8 in 2024-04-14T15:26:00.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch apply iscsi foo u p' 2024-04-14T15:26:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[24869]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 6.7 KiB/s wr, 227 op/s 2024-04-14T15:26:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[24869]: osdmap e77: 8 total, 8 up, 8 in 2024-04-14T15:26:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:26:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:26:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:00.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:00.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[29517]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 6.7 KiB/s wr, 227 op/s 2024-04-14T15:26:00.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[29517]: osdmap e77: 8 total, 8 up, 8 in 2024-04-14T15:26:00.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:00.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:00.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:26:00.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:26:00.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:00.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:00 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:00.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:00 smithi196 ceph-mon[25696]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 6.7 KiB/s wr, 227 op/s 2024-04-14T15:26:00.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:00 smithi196 ceph-mon[25696]: osdmap e77: 8 total, 8 up, 8 in 2024-04-14T15:26:00.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:00 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:00.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:00 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:00.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:00 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:26:00.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:00 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:26:00.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:00 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:00.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:00 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:01.588 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:01 smithi016 ceph-mon[29517]: Saving service rgw.smpl spec with placement count:2 2024-04-14T15:26:01.588 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:01 smithi016 ceph-mon[29517]: Deploying daemon rgw.smpl.smithi016.sjsxjs on smithi016 2024-04-14T15:26:01.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:01 smithi016 ceph-mon[24869]: Saving service rgw.smpl spec with placement count:2 2024-04-14T15:26:01.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:01 smithi016 ceph-mon[24869]: Deploying daemon rgw.smpl.smithi016.sjsxjs on smithi016 2024-04-14T15:26:01.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:01 smithi196 ceph-mon[25696]: Saving service rgw.smpl spec with placement count:2 2024-04-14T15:26:01.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:01 smithi196 ceph-mon[25696]: Deploying daemon rgw.smpl.smithi016.sjsxjs on smithi016 2024-04-14T15:26:02.598 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:26:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:02] "GET /metrics HTTP/1.1" 200 196825 "" "Prometheus/2.33.4" 2024-04-14T15:26:02.598 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:02 smithi016 ceph-mon[29517]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 5.6 KiB/s wr, 189 op/s 2024-04-14T15:26:02.598 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:02 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:02.598 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:02 smithi016 ceph-mon[24869]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 5.6 KiB/s wr, 189 op/s 2024-04-14T15:26:02.598 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:02 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:02.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:02 smithi196 ceph-mon[25696]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 5.6 KiB/s wr, 189 op/s 2024-04-14T15:26:02.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:02 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:03.090 INFO:teuthology.orchestra.run.smithi016.stdout:Scheduled iscsi.foo update... 2024-04-14T15:26:04.003 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'sleep 120' 2024-04-14T15:26:04.317 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[29517]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:26:04.317 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[29517]: Saving service iscsi.foo spec with placement count:1 2024-04-14T15:26:04.317 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:04.317 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[29517]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 6.0 KiB/s wr, 297 op/s 2024-04-14T15:26:04.317 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:04.317 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:26:04.317 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:26:04.317 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:04.317 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:04.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[24869]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:26:04.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[24869]: Saving service iscsi.foo spec with placement count:1 2024-04-14T15:26:04.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:04.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[24869]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 6.0 KiB/s wr, 297 op/s 2024-04-14T15:26:04.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:04.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:26:04.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:26:04.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:04.318 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:04 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:04 smithi196 ceph-mon[25696]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:26:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:04 smithi196 ceph-mon[25696]: Saving service iscsi.foo spec with placement count:1 2024-04-14T15:26:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:04 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:04 smithi196 ceph-mon[25696]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 6.0 KiB/s wr, 297 op/s 2024-04-14T15:26:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:04 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:04 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:26:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:04 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T15:26:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:04 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:04 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:05.214 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:05 smithi016 ceph-mon[29517]: Deploying daemon rgw.smpl.smithi196.uwpszz on smithi196 2024-04-14T15:26:05.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:05 smithi016 ceph-mon[24869]: Deploying daemon rgw.smpl.smithi196.uwpszz on smithi196 2024-04-14T15:26:05.353 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:05 smithi196 ceph-mon[25696]: Deploying daemon rgw.smpl.smithi196.uwpszz on smithi196 2024-04-14T15:26:06.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:06 smithi016 ceph-mon[29517]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s rd, 4.5 KiB/s wr, 268 op/s 2024-04-14T15:26:06.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:06 smithi016 ceph-mon[24869]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s rd, 4.5 KiB/s wr, 268 op/s 2024-04-14T15:26:06.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:06 smithi196 ceph-mon[25696]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s rd, 4.5 KiB/s wr, 268 op/s 2024-04-14T15:26:07.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:07.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:07.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:07.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:07.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:07.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:07.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:07.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:07.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:08.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-04-14T15:26:08.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-04-14T15:26:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-04-14T15:26:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-14T15:26:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:08 smithi196 ceph-mon[25696]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 344 B/s wr, 112 op/s 2024-04-14T15:26:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:26:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:26:08.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:08 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:08.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-04-14T15:26:08.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-04-14T15:26:08.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-04-14T15:26:08.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-14T15:26:08.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[24869]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 344 B/s wr, 112 op/s 2024-04-14T15:26:08.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:26:08.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:26:08.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:08.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-04-14T15:26:08.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-04-14T15:26:08.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-04-14T15:26:08.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-14T15:26:08.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[29517]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 344 B/s wr, 112 op/s 2024-04-14T15:26:08.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:26:08.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:26:08.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:08 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:09.414 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:26:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:26:09.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:09 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-04-14T15:26:09.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:09 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-04-14T15:26:09.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:09 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-04-14T15:26:09.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:09 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-14T15:26:09.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:09 smithi196 ceph-mon[25696]: osdmap e78: 8 total, 8 up, 8 in 2024-04-14T15:26:09.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:09 smithi196 ceph-mon[25696]: osdmap e79: 8 total, 8 up, 8 in 2024-04-14T15:26:09.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:09 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-04-14T15:26:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-04-14T15:26:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-04-14T15:26:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-14T15:26:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[24869]: osdmap e78: 8 total, 8 up, 8 in 2024-04-14T15:26:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[24869]: osdmap e79: 8 total, 8 up, 8 in 2024-04-14T15:26:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:09.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-04-14T15:26:09.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-04-14T15:26:09.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-04-14T15:26:09.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-14T15:26:09.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[29517]: osdmap e78: 8 total, 8 up, 8 in 2024-04-14T15:26:09.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[29517]: osdmap e79: 8 total, 8 up, 8 in 2024-04-14T15:26:09.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:09 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[24869]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 243 KiB/s rd, 639 B/s wr, 376 op/s 2024-04-14T15:26:10.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:26:10.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-14T15:26:10.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:10.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[29517]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 243 KiB/s rd, 639 B/s wr, 376 op/s 2024-04-14T15:26:10.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:26:10.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-14T15:26:10.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:10 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:10.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:10 smithi196 ceph-mon[25696]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 243 KiB/s rd, 639 B/s wr, 376 op/s 2024-04-14T15:26:10.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:10 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:10 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:10 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:10 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:10.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:10 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:26:10.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:10 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-14T15:26:10.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:10 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:11.537 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:11 smithi016 ceph-mon[29517]: Checking dashboard <-> RGW credentials 2024-04-14T15:26:11.537 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:11 smithi016 ceph-mon[29517]: Checking pool "foo" exists for service iscsi.foo 2024-04-14T15:26:11.537 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:11 smithi016 ceph-mon[29517]: Deploying daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:26:11.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:11 smithi016 ceph-mon[24869]: Checking dashboard <-> RGW credentials 2024-04-14T15:26:11.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:11 smithi016 ceph-mon[24869]: Checking pool "foo" exists for service iscsi.foo 2024-04-14T15:26:11.537 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:11 smithi016 ceph-mon[24869]: Deploying daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:26:11.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:11 smithi196 ceph-mon[25696]: Checking dashboard <-> RGW credentials 2024-04-14T15:26:11.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:11 smithi196 ceph-mon[25696]: Checking pool "foo" exists for service iscsi.foo 2024-04-14T15:26:11.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:11 smithi196 ceph-mon[25696]: Deploying daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:26:12.471 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:12 smithi016 ceph-mon[24869]: mgrmap e21: y(active, since 2m), standbys: x 2024-04-14T15:26:12.471 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:12 smithi016 ceph-mon[24869]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 255 B/s wr, 293 op/s 2024-04-14T15:26:12.472 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:26:12 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:12] "GET /metrics HTTP/1.1" 200 213284 "" "Prometheus/2.33.4" 2024-04-14T15:26:12.472 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:12 smithi016 ceph-mon[29517]: mgrmap e21: y(active, since 2m), standbys: x 2024-04-14T15:26:12.472 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:12 smithi016 ceph-mon[29517]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 255 B/s wr, 293 op/s 2024-04-14T15:26:12.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:12 smithi196 ceph-mon[25696]: mgrmap e21: y(active, since 2m), standbys: x 2024-04-14T15:26:12.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:12 smithi196 ceph-mon[25696]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 255 B/s wr, 293 op/s 2024-04-14T15:26:14.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:14.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:14.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:14.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[29517]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 639 B/s wr, 257 op/s 2024-04-14T15:26:14.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3469934983' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:26:14.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1121913526' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2467795236"}]: dispatch 2024-04-14T15:26:14.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2467795236"}]: dispatch 2024-04-14T15:26:14.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:14.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:14.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:14.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[24869]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 639 B/s wr, 257 op/s 2024-04-14T15:26:14.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3469934983' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:26:14.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1121913526' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2467795236"}]: dispatch 2024-04-14T15:26:14.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:14 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2467795236"}]: dispatch 2024-04-14T15:26:14.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:14 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:14.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:14 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:14.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:14 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:14.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:14 smithi196 ceph-mon[25696]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 639 B/s wr, 257 op/s 2024-04-14T15:26:14.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:14 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/3469934983' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:26:14.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:14 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1121913526' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2467795236"}]: dispatch 2024-04-14T15:26:14.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:14 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2467795236"}]: dispatch 2024-04-14T15:26:16.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:15 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2467795236"}]': finished 2024-04-14T15:26:16.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:15 smithi196 ceph-mon[25696]: osdmap e80: 8 total, 8 up, 8 in 2024-04-14T15:26:16.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:15 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/420466103' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2899000330"}]: dispatch 2024-04-14T15:26:16.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:15 smithi196 ceph-mon[25696]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 724 B/s wr, 291 op/s 2024-04-14T15:26:16.097 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:15 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2467795236"}]': finished 2024-04-14T15:26:16.097 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:15 smithi016 ceph-mon[29517]: osdmap e80: 8 total, 8 up, 8 in 2024-04-14T15:26:16.098 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:15 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/420466103' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2899000330"}]: dispatch 2024-04-14T15:26:16.098 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:15 smithi016 ceph-mon[29517]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 724 B/s wr, 291 op/s 2024-04-14T15:26:16.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:15 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/2467795236"}]': finished 2024-04-14T15:26:16.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:15 smithi016 ceph-mon[24869]: osdmap e80: 8 total, 8 up, 8 in 2024-04-14T15:26:16.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:15 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/420466103' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2899000330"}]: dispatch 2024-04-14T15:26:16.098 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:15 smithi016 ceph-mon[24869]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 724 B/s wr, 291 op/s 2024-04-14T15:26:17.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/420466103' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2899000330"}]': finished 2024-04-14T15:26:17.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: osdmap e81: 8 total, 8 up, 8 in 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/528446771' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/304800256"}]: dispatch 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/304800256"}]: dispatch 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:17.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:17.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/420466103' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2899000330"}]': finished 2024-04-14T15:26:17.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: osdmap e81: 8 total, 8 up, 8 in 2024-04-14T15:26:17.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/528446771' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/304800256"}]: dispatch 2024-04-14T15:26:17.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/304800256"}]: dispatch 2024-04-14T15:26:17.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T15:26:17.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T15:26:17.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-04-14T15:26:17.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:17.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:17.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/420466103' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2899000330"}]': finished 2024-04-14T15:26:17.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: osdmap e81: 8 total, 8 up, 8 in 2024-04-14T15:26:17.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/528446771' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/304800256"}]: dispatch 2024-04-14T15:26:17.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/304800256"}]: dispatch 2024-04-14T15:26:17.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T15:26:17.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T15:26:17.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-04-14T15:26:17.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:17.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:26:17.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:26:17.987 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:17 smithi196 ceph-mon[25696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T15:26:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:17 smithi196 ceph-mon[25696]: Adding iSCSI gateway http://:@172.21.15.16:5000 to Dashboard 2024-04-14T15:26:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:17 smithi196 ceph-mon[25696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T15:26:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:17 smithi196 ceph-mon[25696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-04-14T15:26:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:17 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/304800256"}]': finished 2024-04-14T15:26:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:17 smithi196 ceph-mon[25696]: osdmap e82: 8 total, 8 up, 8 in 2024-04-14T15:26:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:17 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/4224311824' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4034272136"}]: dispatch 2024-04-14T15:26:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:17 smithi196 ceph-mon[25696]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-14T15:26:17.988 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:17 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[29517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T15:26:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[29517]: Adding iSCSI gateway http://:@172.21.15.16:5000 to Dashboard 2024-04-14T15:26:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[29517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T15:26:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[29517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-04-14T15:26:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/304800256"}]': finished 2024-04-14T15:26:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[29517]: osdmap e82: 8 total, 8 up, 8 in 2024-04-14T15:26:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4224311824' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4034272136"}]: dispatch 2024-04-14T15:26:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[29517]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-14T15:26:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[24869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T15:26:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[24869]: Adding iSCSI gateway http://:@172.21.15.16:5000 to Dashboard 2024-04-14T15:26:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[24869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T15:26:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[24869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi016"}]: dispatch 2024-04-14T15:26:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/304800256"}]': finished 2024-04-14T15:26:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[24869]: osdmap e82: 8 total, 8 up, 8 in 2024-04-14T15:26:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4224311824' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4034272136"}]: dispatch 2024-04-14T15:26:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[24869]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-14T15:26:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:17 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:18.839 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:18 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4224311824' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4034272136"}]': finished 2024-04-14T15:26:18.839 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:18 smithi016 ceph-mon[24869]: osdmap e83: 8 total, 8 up, 8 in 2024-04-14T15:26:18.839 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:18 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2484303472' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/506900340"}]: dispatch 2024-04-14T15:26:18.839 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:18 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4224311824' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4034272136"}]': finished 2024-04-14T15:26:18.839 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:18 smithi016 ceph-mon[29517]: osdmap e83: 8 total, 8 up, 8 in 2024-04-14T15:26:18.841 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:18 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2484303472' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/506900340"}]: dispatch 2024-04-14T15:26:18.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:18 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/4224311824' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4034272136"}]': finished 2024-04-14T15:26:18.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:18 smithi196 ceph-mon[25696]: osdmap e83: 8 total, 8 up, 8 in 2024-04-14T15:26:18.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:18 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2484303472' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/506900340"}]: dispatch 2024-04-14T15:26:19.576 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:26:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:26:19.876 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:19 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2484303472' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/506900340"}]': finished 2024-04-14T15:26:19.877 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:19 smithi196 ceph-mon[25696]: osdmap e84: 8 total, 8 up, 8 in 2024-04-14T15:26:19.877 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:19 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1745406761' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/37325361"}]: dispatch 2024-04-14T15:26:19.877 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:19 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/37325361"}]: dispatch 2024-04-14T15:26:19.877 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:19 smithi196 ceph-mon[25696]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:20.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:19 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2484303472' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/506900340"}]': finished 2024-04-14T15:26:20.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:19 smithi016 ceph-mon[29517]: osdmap e84: 8 total, 8 up, 8 in 2024-04-14T15:26:20.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:19 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1745406761' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/37325361"}]: dispatch 2024-04-14T15:26:20.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:19 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/37325361"}]: dispatch 2024-04-14T15:26:20.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:19 smithi016 ceph-mon[29517]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:20.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:19 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2484303472' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/506900340"}]': finished 2024-04-14T15:26:20.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:19 smithi016 ceph-mon[24869]: osdmap e84: 8 total, 8 up, 8 in 2024-04-14T15:26:20.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:19 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1745406761' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/37325361"}]: dispatch 2024-04-14T15:26:20.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:19 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/37325361"}]: dispatch 2024-04-14T15:26:20.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:19 smithi016 ceph-mon[24869]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/37325361"}]': finished 2024-04-14T15:26:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[29517]: osdmap e85: 8 total, 8 up, 8 in 2024-04-14T15:26:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4034464804' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/86456086"}]: dispatch 2024-04-14T15:26:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/86456086"}]: dispatch 2024-04-14T15:26:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[29517]: Checking dashboard <-> RGW credentials 2024-04-14T15:26:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:21.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/37325361"}]': finished 2024-04-14T15:26:21.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[24869]: osdmap e85: 8 total, 8 up, 8 in 2024-04-14T15:26:21.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:21.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:21.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4034464804' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/86456086"}]: dispatch 2024-04-14T15:26:21.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/86456086"}]: dispatch 2024-04-14T15:26:21.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[24869]: Checking dashboard <-> RGW credentials 2024-04-14T15:26:21.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:20 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:21.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:20 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/37325361"}]': finished 2024-04-14T15:26:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:20 smithi196 ceph-mon[25696]: osdmap e85: 8 total, 8 up, 8 in 2024-04-14T15:26:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:20 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:20 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:20 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/4034464804' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/86456086"}]: dispatch 2024-04-14T15:26:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:20 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/86456086"}]: dispatch 2024-04-14T15:26:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:20 smithi196 ceph-mon[25696]: Checking dashboard <-> RGW credentials 2024-04-14T15:26:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:20 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:21 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/86456086"}]': finished 2024-04-14T15:26:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:21 smithi196 ceph-mon[25696]: osdmap e86: 8 total, 8 up, 8 in 2024-04-14T15:26:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:21 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2187292871' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2467795236"}]: dispatch 2024-04-14T15:26:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:21 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2467795236"}]: dispatch 2024-04-14T15:26:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:21 smithi196 ceph-mon[25696]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:22.256 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:21 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/86456086"}]': finished 2024-04-14T15:26:22.256 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:21 smithi016 ceph-mon[29517]: osdmap e86: 8 total, 8 up, 8 in 2024-04-14T15:26:22.256 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:21 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2187292871' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2467795236"}]: dispatch 2024-04-14T15:26:22.256 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:21 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2467795236"}]: dispatch 2024-04-14T15:26:22.256 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:21 smithi016 ceph-mon[29517]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:22.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:21 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/86456086"}]': finished 2024-04-14T15:26:22.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:21 smithi016 ceph-mon[24869]: osdmap e86: 8 total, 8 up, 8 in 2024-04-14T15:26:22.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:21 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2187292871' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2467795236"}]: dispatch 2024-04-14T15:26:22.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:21 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2467795236"}]: dispatch 2024-04-14T15:26:22.257 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:21 smithi016 ceph-mon[24869]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:22.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:26:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:22] "GET /metrics HTTP/1.1" 200 214892 "" "Prometheus/2.33.4" 2024-04-14T15:26:23.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:22 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2467795236"}]': finished 2024-04-14T15:26:23.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:22 smithi196 ceph-mon[25696]: osdmap e87: 8 total, 8 up, 8 in 2024-04-14T15:26:23.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:22 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2337651137' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/37325361"}]: dispatch 2024-04-14T15:26:23.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:22 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:23.204 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:22 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2467795236"}]': finished 2024-04-14T15:26:23.204 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:22 smithi016 ceph-mon[29517]: osdmap e87: 8 total, 8 up, 8 in 2024-04-14T15:26:23.204 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:22 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2337651137' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/37325361"}]: dispatch 2024-04-14T15:26:23.204 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:22 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:23.205 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:22 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/2467795236"}]': finished 2024-04-14T15:26:23.205 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:22 smithi016 ceph-mon[24869]: osdmap e87: 8 total, 8 up, 8 in 2024-04-14T15:26:23.205 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:22 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2337651137' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/37325361"}]: dispatch 2024-04-14T15:26:23.205 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:22 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:26:23.861 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:26:23.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs" 2024-04-14T15:26:23.861 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:26:23.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:26:24.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:23 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2337651137' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/37325361"}]': finished 2024-04-14T15:26:24.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:23 smithi196 ceph-mon[25696]: osdmap e88: 8 total, 8 up, 8 in 2024-04-14T15:26:24.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:23 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:24.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:23 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2070976862' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4006149639"}]: dispatch 2024-04-14T15:26:24.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:23 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4006149639"}]: dispatch 2024-04-14T15:26:24.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:23 smithi196 ceph-mon[25696]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T15:26:24.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2337651137' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/37325361"}]': finished 2024-04-14T15:26:24.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[24869]: osdmap e88: 8 total, 8 up, 8 in 2024-04-14T15:26:24.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:24.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2070976862' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4006149639"}]: dispatch 2024-04-14T15:26:24.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4006149639"}]: dispatch 2024-04-14T15:26:24.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[24869]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T15:26:24.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2337651137' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/37325361"}]': finished 2024-04-14T15:26:24.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[29517]: osdmap e88: 8 total, 8 up, 8 in 2024-04-14T15:26:24.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:24.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2070976862' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4006149639"}]: dispatch 2024-04-14T15:26:24.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4006149639"}]: dispatch 2024-04-14T15:26:24.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:23 smithi016 ceph-mon[29517]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T15:26:25.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:24 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4006149639"}]': finished 2024-04-14T15:26:25.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:24 smithi196 ceph-mon[25696]: osdmap e89: 8 total, 8 up, 8 in 2024-04-14T15:26:25.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:24 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1679609319' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/910690549"}]: dispatch 2024-04-14T15:26:25.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:24 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4006149639"}]': finished 2024-04-14T15:26:25.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:24 smithi016 ceph-mon[29517]: osdmap e89: 8 total, 8 up, 8 in 2024-04-14T15:26:25.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:24 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1679609319' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/910690549"}]: dispatch 2024-04-14T15:26:25.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:24 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/4006149639"}]': finished 2024-04-14T15:26:25.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:24 smithi016 ceph-mon[24869]: osdmap e89: 8 total, 8 up, 8 in 2024-04-14T15:26:25.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:24 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1679609319' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/910690549"}]: dispatch 2024-04-14T15:26:26.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:25 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1679609319' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/910690549"}]': finished 2024-04-14T15:26:26.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:25 smithi196 ceph-mon[25696]: osdmap e90: 8 total, 8 up, 8 in 2024-04-14T15:26:26.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:25 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/964605153' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/646505488"}]: dispatch 2024-04-14T15:26:26.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:25 smithi196 ceph-mon[25696]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T15:26:26.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:25 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1679609319' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/910690549"}]': finished 2024-04-14T15:26:26.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:25 smithi016 ceph-mon[29517]: osdmap e90: 8 total, 8 up, 8 in 2024-04-14T15:26:26.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:25 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/964605153' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/646505488"}]: dispatch 2024-04-14T15:26:26.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:25 smithi016 ceph-mon[29517]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T15:26:26.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:25 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1679609319' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/910690549"}]': finished 2024-04-14T15:26:26.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:25 smithi016 ceph-mon[24869]: osdmap e90: 8 total, 8 up, 8 in 2024-04-14T15:26:26.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:25 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/964605153' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/646505488"}]: dispatch 2024-04-14T15:26:26.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:25 smithi016 ceph-mon[24869]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T15:26:27.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:26 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/964605153' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/646505488"}]': finished 2024-04-14T15:26:27.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:26 smithi016 ceph-mon[24869]: osdmap e91: 8 total, 8 up, 8 in 2024-04-14T15:26:27.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:26 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3196561618' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2585190683"}]: dispatch 2024-04-14T15:26:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:26 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2585190683"}]: dispatch 2024-04-14T15:26:27.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:26 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/964605153' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/646505488"}]': finished 2024-04-14T15:26:27.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:26 smithi016 ceph-mon[29517]: osdmap e91: 8 total, 8 up, 8 in 2024-04-14T15:26:27.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:26 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3196561618' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2585190683"}]: dispatch 2024-04-14T15:26:27.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:26 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2585190683"}]: dispatch 2024-04-14T15:26:27.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:26 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/964605153' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/646505488"}]': finished 2024-04-14T15:26:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:26 smithi196 ceph-mon[25696]: osdmap e91: 8 total, 8 up, 8 in 2024-04-14T15:26:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:26 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/3196561618' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2585190683"}]: dispatch 2024-04-14T15:26:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:26 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2585190683"}]: dispatch 2024-04-14T15:26:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:27 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2585190683"}]': finished 2024-04-14T15:26:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:27 smithi196 ceph-mon[25696]: osdmap e92: 8 total, 8 up, 8 in 2024-04-14T15:26:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:27 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/662877758' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1366879040"}]: dispatch 2024-04-14T15:26:28.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:27 smithi196 ceph-mon[25696]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:26:28.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:27 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2585190683"}]': finished 2024-04-14T15:26:28.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:27 smithi016 ceph-mon[24869]: osdmap e92: 8 total, 8 up, 8 in 2024-04-14T15:26:28.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:27 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/662877758' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1366879040"}]: dispatch 2024-04-14T15:26:28.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:27 smithi016 ceph-mon[24869]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:26:28.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:27 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2585190683"}]': finished 2024-04-14T15:26:28.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:27 smithi016 ceph-mon[29517]: osdmap e92: 8 total, 8 up, 8 in 2024-04-14T15:26:28.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:27 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/662877758' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1366879040"}]: dispatch 2024-04-14T15:26:28.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:27 smithi016 ceph-mon[29517]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:26:29.305 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:28 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/662877758' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1366879040"}]': finished 2024-04-14T15:26:29.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:28 smithi196 ceph-mon[25696]: osdmap e93: 8 total, 8 up, 8 in 2024-04-14T15:26:29.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:28 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/819090361' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4006149639"}]: dispatch 2024-04-14T15:26:29.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:28 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4006149639"}]: dispatch 2024-04-14T15:26:29.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:28 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/662877758' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1366879040"}]': finished 2024-04-14T15:26:29.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:28 smithi016 ceph-mon[24869]: osdmap e93: 8 total, 8 up, 8 in 2024-04-14T15:26:29.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:28 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/819090361' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4006149639"}]: dispatch 2024-04-14T15:26:29.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:28 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4006149639"}]: dispatch 2024-04-14T15:26:29.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:28 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/662877758' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1366879040"}]': finished 2024-04-14T15:26:29.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:28 smithi016 ceph-mon[29517]: osdmap e93: 8 total, 8 up, 8 in 2024-04-14T15:26:29.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:28 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/819090361' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4006149639"}]: dispatch 2024-04-14T15:26:29.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:28 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4006149639"}]: dispatch 2024-04-14T15:26:29.558 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:26:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:26:30.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:29 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4006149639"}]': finished 2024-04-14T15:26:30.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:29 smithi016 ceph-mon[24869]: osdmap e94: 8 total, 8 up, 8 in 2024-04-14T15:26:30.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:29 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3058622299' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3638250575"}]: dispatch 2024-04-14T15:26:30.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:29 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3638250575"}]: dispatch 2024-04-14T15:26:30.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:29 smithi016 ceph-mon[24869]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:30.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:29 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4006149639"}]': finished 2024-04-14T15:26:30.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:29 smithi016 ceph-mon[29517]: osdmap e94: 8 total, 8 up, 8 in 2024-04-14T15:26:30.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:29 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3058622299' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3638250575"}]: dispatch 2024-04-14T15:26:30.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:29 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3638250575"}]: dispatch 2024-04-14T15:26:30.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:29 smithi016 ceph-mon[29517]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:30.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:29 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/4006149639"}]': finished 2024-04-14T15:26:30.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:29 smithi196 ceph-mon[25696]: osdmap e94: 8 total, 8 up, 8 in 2024-04-14T15:26:30.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:29 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/3058622299' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3638250575"}]: dispatch 2024-04-14T15:26:30.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:29 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3638250575"}]: dispatch 2024-04-14T15:26:30.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:29 smithi196 ceph-mon[25696]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:31.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:30 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3638250575"}]': finished 2024-04-14T15:26:31.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:30 smithi016 ceph-mon[24869]: osdmap e95: 8 total, 8 up, 8 in 2024-04-14T15:26:31.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:30 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3638250575"}]': finished 2024-04-14T15:26:31.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:30 smithi016 ceph-mon[29517]: osdmap e95: 8 total, 8 up, 8 in 2024-04-14T15:26:31.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:30 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3638250575"}]': finished 2024-04-14T15:26:31.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:30 smithi196 ceph-mon[25696]: osdmap e95: 8 total, 8 up, 8 in 2024-04-14T15:26:32.314 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:31 smithi016 ceph-mon[29517]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:26:32.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:31 smithi016 ceph-mon[24869]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:26:32.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:31 smithi196 ceph-mon[25696]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:26:32.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:26:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:32] "GET /metrics HTTP/1.1" 200 214892 "" "Prometheus/2.33.4" 2024-04-14T15:26:33.860 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:26:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=error ts=2024-04-14T15:26:33.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:26:33.861 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:26:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:26:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:26:33.861 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:26:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:26:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs" 2024-04-14T15:26:34.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:34 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:34.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:34 smithi016 ceph-mon[29517]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:26:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:34 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:34 smithi016 ceph-mon[24869]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:26:34.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:34 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:34.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:34 smithi196 ceph-mon[25696]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T15:26:36.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:36 smithi016 ceph-mon[24869]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:26:36.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:36 smithi016 ceph-mon[29517]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:26:36.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:36 smithi196 ceph-mon[25696]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:26:38.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:38 smithi016 ceph-mon[29517]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-04-14T15:26:38.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:38 smithi016 ceph-mon[24869]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-04-14T15:26:38.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:38 smithi196 ceph-mon[25696]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-04-14T15:26:39.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:26:39 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:26:40.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:39 smithi016 ceph-mon[29517]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:26:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:39 smithi016 ceph-mon[24869]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:26:40.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:39 smithi196 ceph-mon[25696]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:26:42.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:26:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:42] "GET /metrics HTTP/1.1" 200 214905 "" "Prometheus/2.33.4" 2024-04-14T15:26:42.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:42 smithi016 ceph-mon[29517]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-04-14T15:26:42.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:42 smithi016 ceph-mon[24869]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-04-14T15:26:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:42 smithi196 ceph-mon[25696]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-04-14T15:26:43.860 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:26:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=error ts=2024-04-14T15:26:43.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:26:44.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:44 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:44.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:44 smithi016 ceph-mon[24869]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:44.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:44 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:44.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:44 smithi016 ceph-mon[29517]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:44.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:44 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:44.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:44 smithi196 ceph-mon[25696]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:46.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:46 smithi016 ceph-mon[29517]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:26:46.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:46 smithi016 ceph-mon[24869]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:26:46.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:46 smithi196 ceph-mon[25696]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:26:48.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:48 smithi016 ceph-mon[29517]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:26:48.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:48 smithi016 ceph-mon[24869]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:26:48.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:48 smithi196 ceph-mon[25696]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:26:49.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:26:49 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:26:50.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:49 smithi016 ceph-mon[24869]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:49 smithi016 ceph-mon[29517]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:50.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:49 smithi196 ceph-mon[25696]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:52.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:52 smithi016 ceph-mon[24869]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:26:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:52 smithi016 ceph-mon[29517]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:26:52.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:26:52 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:52] "GET /metrics HTTP/1.1" 200 214853 "" "Prometheus/2.33.4" 2024-04-14T15:26:52.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:52 smithi196 ceph-mon[25696]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:26:54.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:54 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:54 smithi016 ceph-mon[29517]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:54 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:54 smithi016 ceph-mon[24869]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:54 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:26:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:54 smithi196 ceph-mon[25696]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:26:56.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:56 smithi016 ceph-mon[29517]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:26:56.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:56 smithi016 ceph-mon[24869]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:26:56.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:56 smithi196 ceph-mon[25696]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:26:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:58 smithi016 ceph-mon[24869]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:26:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:58 smithi016 ceph-mon[29517]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:26:58.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:58 smithi196 ceph-mon[25696]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:26:59.634 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:26:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:26:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:26:59.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:26:59 smithi196 ceph-mon[25696]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:00.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:26:59 smithi016 ceph-mon[24869]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:00.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:26:59 smithi016 ceph-mon[29517]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:02.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:02 smithi016 ceph-mon[29517]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:02.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:02 smithi016 ceph-mon[24869]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:02.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:27:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:02] "GET /metrics HTTP/1.1" 200 214853 "" "Prometheus/2.33.4" 2024-04-14T15:27:02.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:02 smithi196 ceph-mon[25696]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:04.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:04 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:04 smithi016 ceph-mon[24869]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:04 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:04 smithi016 ceph-mon[29517]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:04.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:04 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:04.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:04 smithi196 ceph-mon[25696]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:06.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:06 smithi016 ceph-mon[29517]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:06.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:06 smithi016 ceph-mon[24869]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:06.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:06 smithi196 ceph-mon[25696]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:07.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:27:07.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:27:07.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:27:07.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:27:07.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:27:07.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:27:08.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:08 smithi016 ceph-mon[29517]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:08.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:08 smithi016 ceph-mon[24869]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:08.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:08 smithi196 ceph-mon[25696]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:09.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:27:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:27:10.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:09 smithi016 ceph-mon[29517]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:10.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:09 smithi016 ceph-mon[24869]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:10.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:09 smithi196 ceph-mon[25696]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:12.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:12 smithi016 ceph-mon[24869]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:12.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:12 smithi016 ceph-mon[29517]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:12.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:27:12 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:12] "GET /metrics HTTP/1.1" 200 214821 "" "Prometheus/2.33.4" 2024-04-14T15:27:12.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:12 smithi196 ceph-mon[25696]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:14.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:14 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:14.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:14 smithi016 ceph-mon[29517]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:14.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:14 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:14.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:14 smithi016 ceph-mon[24869]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:14.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:14 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:14.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:14 smithi196 ceph-mon[25696]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:16.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:16 smithi016 ceph-mon[29517]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:16.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:16 smithi016 ceph-mon[24869]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:16.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:16 smithi196 ceph-mon[25696]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:18.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:18 smithi016 ceph-mon[29517]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:18.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:18 smithi016 ceph-mon[24869]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:18.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:18 smithi196 ceph-mon[25696]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:19.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:27:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:27:20.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:19 smithi016 ceph-mon[29517]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:20.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:19 smithi016 ceph-mon[24869]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:20.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:19 smithi196 ceph-mon[25696]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:20.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:20 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:27:20.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:20 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:27:20.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:20 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:27:21.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:20 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:27:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:20 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:27:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:20 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:27:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:20 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:27:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:20 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:27:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:20 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:27:22.012 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:21 smithi016 ceph-mon[29517]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:22.012 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:21 smithi016 ceph-mon[24869]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:22.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:21 smithi196 ceph-mon[25696]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:22.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:27:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:22] "GET /metrics HTTP/1.1" 200 214813 "" "Prometheus/2.33.4" 2024-04-14T15:27:24.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:24 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:24.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:24 smithi016 ceph-mon[29517]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:24.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:24 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:24.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:24 smithi016 ceph-mon[24869]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:24.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:24 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:24.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:24 smithi196 ceph-mon[25696]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:26.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:26 smithi016 ceph-mon[24869]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:26.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:26 smithi016 ceph-mon[29517]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:26.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:26 smithi196 ceph-mon[25696]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:28.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:28 smithi016 ceph-mon[29517]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:28.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:28 smithi016 ceph-mon[24869]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:28.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:28 smithi196 ceph-mon[25696]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:29.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:27:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:27:30.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:29 smithi016 ceph-mon[24869]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:30.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:29 smithi016 ceph-mon[29517]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:30.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:29 smithi196 ceph-mon[25696]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:32.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:32 smithi016 ceph-mon[24869]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:32.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:32 smithi016 ceph-mon[29517]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:32.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:27:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:32] "GET /metrics HTTP/1.1" 200 214813 "" "Prometheus/2.33.4" 2024-04-14T15:27:32.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:32 smithi196 ceph-mon[25696]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:34.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:34 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:34.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:34 smithi016 ceph-mon[29517]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:34 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:34 smithi016 ceph-mon[24869]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:34.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:34 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:34.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:34 smithi196 ceph-mon[25696]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:36.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:36 smithi016 ceph-mon[29517]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:36.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:36 smithi016 ceph-mon[24869]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:36.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:36 smithi196 ceph-mon[25696]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:38.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:38 smithi016 ceph-mon[29517]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:38.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:38 smithi016 ceph-mon[24869]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:38.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:38 smithi196 ceph-mon[25696]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:39.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:27:39 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:27:40.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:39 smithi016 ceph-mon[29517]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:39 smithi016 ceph-mon[24869]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:40.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:39 smithi196 ceph-mon[25696]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:42.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:42 smithi016 ceph-mon[29517]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:42.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:42 smithi016 ceph-mon[24869]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:42.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:27:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:42] "GET /metrics HTTP/1.1" 200 214813 "" "Prometheus/2.33.4" 2024-04-14T15:27:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:42 smithi196 ceph-mon[25696]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:44.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:44 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:44.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:44 smithi016 ceph-mon[24869]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:44.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:44 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:44.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:44 smithi016 ceph-mon[29517]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:44.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:44 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:44.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:44 smithi196 ceph-mon[25696]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:46.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:46 smithi016 ceph-mon[29517]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:46.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:46 smithi016 ceph-mon[24869]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:46.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:46 smithi196 ceph-mon[25696]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:48.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:48 smithi016 ceph-mon[29517]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:48.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:48 smithi016 ceph-mon[24869]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:48.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:48 smithi196 ceph-mon[25696]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:49.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:27:49 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:27:50.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:49 smithi016 ceph-mon[29517]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:49 smithi016 ceph-mon[24869]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:50.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:49 smithi196 ceph-mon[25696]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:52.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:52 smithi016 ceph-mon[29517]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:52.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:52 smithi016 ceph-mon[24869]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:52.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:27:52 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:52] "GET /metrics HTTP/1.1" 200 214820 "" "Prometheus/2.33.4" 2024-04-14T15:27:52.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:52 smithi196 ceph-mon[25696]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:54.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:54 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:54 smithi016 ceph-mon[29517]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:54 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:54 smithi016 ceph-mon[24869]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:54.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:54 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:27:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:54 smithi196 ceph-mon[25696]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:27:56.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:56 smithi016 ceph-mon[29517]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:56.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:56 smithi016 ceph-mon[24869]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:56.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:56 smithi196 ceph-mon[25696]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:58 smithi016 ceph-mon[29517]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:58 smithi016 ceph-mon[24869]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:58 smithi196 ceph-mon[25696]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:27:59.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:27:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:27:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:28:00.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:27:59 smithi016 ceph-mon[29517]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:27:59 smithi016 ceph-mon[24869]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:00.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:27:59 smithi196 ceph-mon[25696]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:02.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:28:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:02] "GET /metrics HTTP/1.1" 200 214820 "" "Prometheus/2.33.4" 2024-04-14T15:28:02.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:02 smithi016 ceph-mon[29517]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:02.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:02 smithi016 ceph-mon[24869]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:02.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:02 smithi196 ceph-mon[25696]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:04.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:04 smithi016 ceph-mon[24869]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:04 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:04 smithi016 ceph-mon[29517]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:04 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:04.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:04 smithi196 ceph-mon[25696]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:04.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:04 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:06.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-14T15:28:06.586 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:06 smithi016 ceph-mon[29517]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:06.587 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:06 smithi016 ceph-mon[24869]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:06.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:06 smithi196 ceph-mon[25696]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:07.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:28:07.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:28:07.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:28:07.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:28:07.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:28:07.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:28:08.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:08 smithi196 ceph-mon[25696]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:08.778 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:08 smithi016 ceph-mon[24869]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:08.779 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:08 smithi016 ceph-mon[29517]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:08.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-14T15:28:09.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:28:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:28:10.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:09 smithi016 ceph-mon[29517]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:10.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:09 smithi016 ceph-mon[24869]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:10.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:09 smithi196 ceph-mon[25696]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:11.553 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-14T15:28:12.510 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:12 smithi016 ceph-mon[29517]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:12.510 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:12 smithi016 ceph-mon[24869]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:12.510 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:28:12 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:12] "GET /metrics HTTP/1.1" 200 214829 "" "Prometheus/2.33.4" 2024-04-14T15:28:12.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:12 smithi196 ceph-mon[25696]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:13.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-14T15:28:14.605 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:14 smithi016 ceph-mon[29517]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:14.605 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:14 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:14.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:14 smithi016 ceph-mon[24869]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:14.606 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:14 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:14.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:14 smithi196 ceph-mon[25696]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:14.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:14 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:15.957 INFO:teuthology.orchestra.run.smithi016.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:28:16.531 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T15:28:16.541 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-14T15:28:16.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-14T15:28:16.567 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:16 smithi016 ceph-mon[29517]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:16.567 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:16.567 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:28:16.567 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:28:16.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:16 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:16.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:16 smithi016 ceph-mon[24869]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:16.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:16.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:28:16.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:28:16.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:16 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:16.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:16 smithi196 ceph-mon[25696]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:16.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:16.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:28:16.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:28:16.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:16 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:17.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:17 smithi196 ceph-mon[25696]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:17 smithi196 ceph-mon[25696]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:28:17.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:17 smithi196 ceph-mon[25696]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:28:17.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:17 smithi016 ceph-mon[29517]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:17.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:17 smithi016 ceph-mon[29517]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:28:17.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:17 smithi016 ceph-mon[29517]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:28:17.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:17 smithi016 ceph-mon[24869]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:17.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:17 smithi016 ceph-mon[24869]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:28:17.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:17 smithi016 ceph-mon[24869]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:28:18.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:18 smithi016 ceph-mon[24869]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:18.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:18 smithi016 ceph-mon[29517]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:18.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:18 smithi196 ceph-mon[25696]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:18.739 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:28:19.434 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (2m) 119s ago 3m 15.4M - ba2b418f427c e0503939608c 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (2m) 119s ago 2m 49.6M - 8.3.5 dad864ee21e9 6cd5c62d1ec5 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (2m) 119s ago 2m 49.4M - 3.5 e1d6a67b021e e2c6c6ed6283 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443 running (7m) 119s ago 7m 411M - 17.2.0 e1d6a67b021e 728647a3cffb 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:9283 running (9m) 119s ago 9m 456M - 17.2.0 e1d6a67b021e 8678f197809c 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (9m) 119s ago 9m 48.4M 2048M 17.2.0 e1d6a67b021e e8cfd2da2fd9 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (7m) 119s ago 7m 39.2M 2048M 17.2.0 e1d6a67b021e f933c5fa37fd 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (8m) 119s ago 8m 39.0M 2048M 17.2.0 e1d6a67b021e 8629057bb8e9 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (3m) 119s ago 3m 13.9M - 1dbe0e931976 568fff30d221 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (3m) 119s ago 3m 20.5M - 1dbe0e931976 142e5a468751 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (7m) 119s ago 7m 49.7M 4017M 17.2.0 e1d6a67b021e b812f3e0fa81 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (6m) 119s ago 6m 48.9M 4017M 17.2.0 e1d6a67b021e c275d1a0eaeb 2024-04-14T15:28:19.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (6m) 119s ago 6m 47.7M 4017M 17.2.0 e1d6a67b021e 592ce301a74a 2024-04-14T15:28:19.436 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (6m) 119s ago 6m 47.0M 4017M 17.2.0 e1d6a67b021e 1ed8730c3fc1 2024-04-14T15:28:19.436 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (5m) 119s ago 5m 49.0M 4272M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:28:19.436 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (5m) 119s ago 5m 45.7M 4272M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:28:19.436 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (5m) 119s ago 5m 45.1M 4272M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:28:19.436 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (4m) 119s ago 4m 50.4M 4272M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:28:19.436 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (2m) 119s ago 3m 44.7M - 514e6a882f6e 371540bc5a5a 2024-04-14T15:28:19.436 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (2m) 119s ago 2m 70.3M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:28:19.436 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (2m) 119s ago 2m 70.5M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:28:19.436 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (2m) 119s ago 2m 68.5M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:28:19.436 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (2m) 119s ago 2m 71.2M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:28:19.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:28:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:28:19.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:19 smithi016 ceph-mon[24869]: from='client.24928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:19.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:19 smithi016 ceph-mon[24869]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:19.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:19 smithi016 ceph-mon[24869]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:19.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:19 smithi016 ceph-mon[24869]: from='client.24940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:19.851 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:28:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:28:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T15:28:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:28:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:28:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T15:28:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:28:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:28:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T15:28:19.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:28:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": {}, 2024-04-14T15:28:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:28:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:28:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:28:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:28:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-14T15:28:19.852 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:28:19.852 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:28:20.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:19 smithi016 ceph-mon[29517]: from='client.24928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:20.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:19 smithi016 ceph-mon[29517]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:20.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:19 smithi016 ceph-mon[29517]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:20.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:19 smithi016 ceph-mon[29517]: from='client.24940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:20.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:19 smithi196 ceph-mon[25696]: from='client.24928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:20.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:19 smithi196 ceph-mon[25696]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:20.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:19 smithi196 ceph-mon[25696]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:20.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:19 smithi196 ceph-mon[25696]: from='client.24940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:20.194 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:28:20.195 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:28:20.195 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:28:20.195 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-04-14T15:28:20.195 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "", 2024-04-14T15:28:20.195 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 image" 2024-04-14T15:28:20.195 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:28:20.591 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-04-14T15:28:20.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:20 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3131971473' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:28:20.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:20 smithi016 ceph-mon[29517]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:20.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:20 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3766631024' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:28:20.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:20 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3131971473' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:28:20.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:20 smithi016 ceph-mon[24869]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:20.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:20 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3766631024' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:28:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:20 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/3131971473' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:28:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:20 smithi196 ceph-mon[25696]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:20 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/3766631024' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:28:22.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:21 smithi016 ceph-mon[29517]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:22.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:21 smithi016 ceph-mon[24869]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:22.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:21 smithi196 ceph-mon[25696]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:22.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:28:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:22] "GET /metrics HTTP/1.1" 200 214819 "" "Prometheus/2.33.4" 2024-04-14T15:28:24.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:24 smithi016 ceph-mon[29517]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:24.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:24 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:24.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:24 smithi016 ceph-mon[24869]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:24.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:24 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:24.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:24 smithi196 ceph-mon[25696]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:24.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:24 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:26.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:26 smithi016 ceph-mon[29517]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:26.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:26 smithi016 ceph-mon[24869]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:26.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:26 smithi196 ceph-mon[25696]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:28.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:28 smithi016 ceph-mon[29517]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:28.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:28 smithi016 ceph-mon[24869]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:28.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:28 smithi196 ceph-mon[25696]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:29.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:28:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:28:30.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:29 smithi016 ceph-mon[29517]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:30.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:29 smithi016 ceph-mon[24869]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:30.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:29 smithi196 ceph-mon[25696]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:32.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:32 smithi016 ceph-mon[29517]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:32.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:28:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:32] "GET /metrics HTTP/1.1" 200 214819 "" "Prometheus/2.33.4" 2024-04-14T15:28:32.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:32 smithi016 ceph-mon[24869]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:32.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:32 smithi196 ceph-mon[25696]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:34.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:34 smithi016 ceph-mon[24869]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:34 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:34.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:34 smithi016 ceph-mon[29517]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:34.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:34 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:34.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:34 smithi196 ceph-mon[25696]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:34.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:34 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:36.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:36 smithi196 ceph-mon[25696]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:36.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:36 smithi016 ceph-mon[29517]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:36.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:36 smithi016 ceph-mon[24869]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:38.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:38 smithi196 ceph-mon[25696]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:38.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:38 smithi016 ceph-mon[29517]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:38.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:38 smithi016 ceph-mon[24869]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:39.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:28:39 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:28:40.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:39 smithi016 ceph-mon[29517]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:39 smithi016 ceph-mon[24869]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:40.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:39 smithi196 ceph-mon[25696]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:42.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:42 smithi016 ceph-mon[29517]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:42.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:42 smithi016 ceph-mon[24869]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:42.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:28:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:42] "GET /metrics HTTP/1.1" 200 214767 "" "Prometheus/2.33.4" 2024-04-14T15:28:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:42 smithi196 ceph-mon[25696]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:44.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:44 smithi196 ceph-mon[25696]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:44.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:44 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:44.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:44 smithi016 ceph-mon[29517]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:44.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:44 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:44.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:44 smithi016 ceph-mon[24869]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:44.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:44 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:46.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:46 smithi196 ceph-mon[25696]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:46.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:46 smithi016 ceph-mon[29517]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:46 smithi016 ceph-mon[24869]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:48.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:48 smithi196 ceph-mon[25696]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:48.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:48 smithi016 ceph-mon[29517]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:48.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:48 smithi016 ceph-mon[24869]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:49.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:28:49 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:28:49.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:49.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:49 smithi196 ceph-mon[25696]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (unknown) 2024-04-14T15:28:49.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:49 smithi196 ceph-mon[25696]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T15:28:49.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:28:49.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:49 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:49.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:49 smithi196 ceph-mon[25696]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-14T15:28:49.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:49 smithi196 ceph-mon[25696]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 on smithi196 2024-04-14T15:28:49.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:49 smithi196 ceph-mon[25696]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:49.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:49.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[24869]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (unknown) 2024-04-14T15:28:49.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[24869]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T15:28:49.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:28:49.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:49.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[24869]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-14T15:28:49.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[24869]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 on smithi196 2024-04-14T15:28:49.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[24869]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:49.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:49.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[29517]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (unknown) 2024-04-14T15:28:49.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[29517]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T15:28:49.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:28:49.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:28:49.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[29517]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-14T15:28:49.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[29517]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 on smithi196 2024-04-14T15:28:49.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:49 smithi016 ceph-mon[29517]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:50.926 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:28:51.531 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:28:51.531 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (3m) 2m ago 3m 15.4M - ba2b418f427c e0503939608c 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (3m) 2m ago 3m 49.6M - 8.3.5 dad864ee21e9 6cd5c62d1ec5 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (2m) 2m ago 2m 49.4M - 3.5 e1d6a67b021e e2c6c6ed6283 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e 728647a3cffb 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:9283 running (9m) 2m ago 9m 456M - 17.2.0 e1d6a67b021e 8678f197809c 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (9m) 2m ago 9m 48.4M 2048M 17.2.0 e1d6a67b021e e8cfd2da2fd9 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (8m) 2m ago 8m 39.2M 2048M 17.2.0 e1d6a67b021e f933c5fa37fd 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (8m) 2m ago 8m 39.0M 2048M 17.2.0 e1d6a67b021e 8629057bb8e9 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (4m) 2m ago 4m 13.9M - 1dbe0e931976 568fff30d221 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (4m) 2m ago 4m 20.5M - 1dbe0e931976 142e5a468751 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (7m) 2m ago 7m 49.7M 4017M 17.2.0 e1d6a67b021e b812f3e0fa81 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (7m) 2m ago 7m 48.9M 4017M 17.2.0 e1d6a67b021e c275d1a0eaeb 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (7m) 2m ago 7m 47.7M 4017M 17.2.0 e1d6a67b021e 592ce301a74a 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (6m) 2m ago 6m 47.0M 4017M 17.2.0 e1d6a67b021e 1ed8730c3fc1 2024-04-14T15:28:51.532 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (6m) 2m ago 6m 49.0M 4272M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:28:51.533 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (5m) 2m ago 5m 45.7M 4272M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:28:51.533 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (5m) 2m ago 5m 45.1M 4272M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:28:51.533 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (5m) 2m ago 5m 50.4M 4272M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:28:51.533 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (3m) 2m ago 4m 44.7M - 514e6a882f6e 371540bc5a5a 2024-04-14T15:28:51.533 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (2m) 2m ago 2m 70.3M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:28:51.533 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (2m) 2m ago 2m 70.5M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:28:51.533 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (2m) 2m ago 2m 68.5M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:28:51.533 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (2m) 2m ago 2m 71.2M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:28:51.891 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": {}, 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:28:51.892 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:28:51.893 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-14T15:28:51.893 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:28:51.893 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:28:52.197 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:28:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:28:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:28:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-04-14T15:28:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "0/23 daemons upgraded", 2024-04-14T15:28:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 image on host smithi196" 2024-04-14T15:28:52.198 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:28:52.468 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:28:52 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:52] "GET /metrics HTTP/1.1" 200 214767 "" "Prometheus/2.33.4" 2024-04-14T15:28:52.468 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:52 smithi016 ceph-mon[29517]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:52.468 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:52 smithi016 ceph-mon[29517]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:52.469 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:52 smithi016 ceph-mon[29517]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:52.469 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:52 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2134455688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:28:52.469 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:52 smithi016 ceph-mon[24869]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:52.469 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:52 smithi016 ceph-mon[24869]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:52.469 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:52 smithi016 ceph-mon[24869]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:52.469 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:52 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2134455688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:28:52.570 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-04-14T15:28:52.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:52 smithi196 ceph-mon[25696]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:52.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:52 smithi196 ceph-mon[25696]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:52 smithi196 ceph-mon[25696]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:52 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2134455688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:28:53.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:53 smithi016 ceph-mon[24869]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:53.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:53 smithi016 ceph-mon[24869]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:53.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:53 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2798353622' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:28:53.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:53 smithi016 ceph-mon[29517]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:53.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:53 smithi016 ceph-mon[29517]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:53.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:53 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2798353622' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:28:53.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:53 smithi196 ceph-mon[25696]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:53.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:53 smithi196 ceph-mon[25696]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:28:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:53 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2798353622' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:28:54.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:54 smithi196 ceph-mon[25696]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:54 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:54.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:54 smithi016 ceph-mon[24869]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:54.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:54 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:54.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:54 smithi016 ceph-mon[29517]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:28:54.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:54 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:28:56.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:56 smithi196 ceph-mon[25696]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:56.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:56 smithi016 ceph-mon[24869]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:56.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:56 smithi016 ceph-mon[29517]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:58 smithi196 ceph-mon[25696]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:58.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:58 smithi016 ceph-mon[24869]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:58.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:58 smithi016 ceph-mon[29517]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:28:59.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:28:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:28:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:29:00.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:28:59 smithi016 ceph-mon[24869]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:00.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:28:59 smithi016 ceph-mon[29517]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:00.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:28:59 smithi196 ceph-mon[25696]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:02.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:02 smithi016 ceph-mon[29517]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:02.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:02 smithi016 ceph-mon[24869]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:02.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:29:02] "GET /metrics HTTP/1.1" 200 214760 "" "Prometheus/2.33.4" 2024-04-14T15:29:02.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:02 smithi196 ceph-mon[25696]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:03.610 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:03 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:03.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:29:03.610 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:03 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:03.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs" 2024-04-14T15:29:04.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:04 smithi196 ceph-mon[25696]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:04.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:04 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:04.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:04 smithi016 ceph-mon[29517]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:04.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:04 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:04 smithi016 ceph-mon[24869]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:04 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:06.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:06 smithi196 ceph-mon[25696]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:06.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:06 smithi016 ceph-mon[29517]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:06.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:06 smithi016 ceph-mon[24869]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:07.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:29:07.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:07 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:29:07.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:29:07.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:07 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:29:07.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:29:07.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:07 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:29:08.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:08 smithi196 ceph-mon[25696]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:08.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:08 smithi016 ceph-mon[29517]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:08.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:08 smithi016 ceph-mon[24869]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:09.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[26884]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:29:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:29:10.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:09 smithi016 ceph-mon[29517]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:10.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:09 smithi016 ceph-mon[24869]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:10.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:09 smithi196 ceph-mon[25696]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:12.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:12 smithi016 ceph-mon[29517]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:12 smithi016 ceph-mon[24869]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:12.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:12 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:29:12] "GET /metrics HTTP/1.1" 200 214760 "" "Prometheus/2.33.4" 2024-04-14T15:29:12.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:12 smithi196 ceph-mon[25696]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:13.860 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:13 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=error ts=2024-04-14T15:29:13.505Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:29:13.861 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:13 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:13.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs" 2024-04-14T15:29:13.861 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:13 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:13.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:29:14.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:14 smithi196 ceph-mon[25696]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:14.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:14 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:14.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:14 smithi016 ceph-mon[29517]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:14.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:14 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:14.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:14 smithi016 ceph-mon[24869]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:14.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:14 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:16.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:16 smithi196 ceph-mon[25696]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:16.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:16 smithi016 ceph-mon[29517]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:16.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:16 smithi016 ceph-mon[24869]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:18.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:17 smithi196 ceph-mon[25696]: Upgrade: Updating mgr.x 2024-04-14T15:29:18.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:17 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:18.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:17 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:29:18.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:17 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:29:18.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:17 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:18.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:17 smithi196 ceph-mon[25696]: Deploying daemon mgr.x on smithi196 2024-04-14T15:29:18.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:17 smithi196 ceph-mon[25696]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:18.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[29517]: Upgrade: Updating mgr.x 2024-04-14T15:29:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:29:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:29:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[29517]: Deploying daemon mgr.x on smithi196 2024-04-14T15:29:18.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[29517]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[24869]: Upgrade: Updating mgr.x 2024-04-14T15:29:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:29:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:29:18.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[24869]: Deploying daemon mgr.x on smithi196 2024-04-14T15:29:18.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:17 smithi016 ceph-mon[24869]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:18.408 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:18 smithi196 systemd[1]: Stopping Ceph mgr.x for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:29:18.409 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:18 smithi196 bash[46770]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr.x" found: no such container 2024-04-14T15:29:18.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:18 smithi196 podman[46781]: 2024-04-14 15:29:18.40888332 +0000 UTC m=+0.148118584 container died 728647a3cffb45d60dafb7e81a8f3fb28ea1b74b6c087e5a28f2668c28778738 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, release=754, vendor=Red Hat, Inc., architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, ceph=True, CEPH_POINT_RELEASE=-17.2.0, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-14T15:29:18.933 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:18 smithi196 podman[46781]: 2024-04-14 15:29:18.746064 +0000 UTC m=+0.485299250 container cleanup 728647a3cffb45d60dafb7e81a8f3fb28ea1b74b6c087e5a28f2668c28778738 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, architecture=x86_64, io.openshift.tags=base centos centos-stream, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , version=8, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, vcs-type=git, GIT_CLEAN=True) 2024-04-14T15:29:18.933 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:18 smithi196 bash[46781]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x 2024-04-14T15:29:18.933 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:18 smithi196 podman[46795]: 2024-04-14 15:29:18.930562704 +0000 UTC m=+0.515016830 container remove 728647a3cffb45d60dafb7e81a8f3fb28ea1b74b6c087e5a28f2668c28778738 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, version=8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, vcs-type=git, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, architecture=x86_64, distribution-scope=public) 2024-04-14T15:29:18.933 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:18 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-14T15:29:19.371 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:18 smithi196 bash[46808]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr.x" found: no such container 2024-04-14T15:29:19.371 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:18 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-14T15:29:19.371 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:18 smithi196 systemd[1]: Stopped Ceph mgr.x for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:29:19.371 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:18 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.x.service: Consumed 16.468s CPU time. 2024-04-14T15:29:19.371 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:19 smithi196 systemd[1]: Starting Ceph mgr.x for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:29:19.633 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:19 smithi196 podman[46908]: 2024-04-14 15:29:19.274262266 +0000 UTC m=+0.018228327 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:29:19.633 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:19 smithi196 podman[46908]: 2024-04-14 15:29:19.398584439 +0000 UTC m=+0.142550501 container create ef0a2cd931a529dbe55aba783ab9f49a145eab49e09c4d7e4612ec57eb6264c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403) 2024-04-14T15:29:19.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:19 smithi196 ceph-mon[25696]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:19.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:19 smithi196 podman[46908]: 2024-04-14 15:29:19.63416676 +0000 UTC m=+0.378132821 container init ef0a2cd931a529dbe55aba783ab9f49a145eab49e09c4d7e4612ec57eb6264c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-a971fd5) 2024-04-14T15:29:19.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:19 smithi196 podman[46908]: 2024-04-14 15:29:19.637071033 +0000 UTC m=+0.381037102 container start ef0a2cd931a529dbe55aba783ab9f49a145eab49e09c4d7e4612ec57eb6264c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_BRANCH=HEAD) 2024-04-14T15:29:19.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:19 smithi196 ceph-mgr[46927]: -- 172.21.15.196:0/3242858353 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5605f84045a0 con 0x5605f83d8c00 2024-04-14T15:29:19.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:19 smithi196 bash[46908]: ef0a2cd931a529dbe55aba783ab9f49a145eab49e09c4d7e4612ec57eb6264c6 2024-04-14T15:29:19.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:19 smithi196 systemd[1]: Started Ceph mgr.x for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:29:20.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:19 smithi016 ceph-mon[29517]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:20.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:19 smithi016 ceph-mon[24869]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:20.508 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:20.211+0000 7fa09e36f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:29:20.508 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:20.305+0000 7fa09e36f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:29:20.610 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:20 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:20.164Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.196:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-04-14T15:29:21.073 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:20.979+0000 7fa09e36f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:29:21.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:20 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:21.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:20 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:21.074 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:20 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:29:21.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:20 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:21.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:20 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:21.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:20 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:29:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:20 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:20 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:20 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:29:21.341 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:21.073+0000 7fa09e36f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:29:21.341 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:21.223+0000 7fa09e36f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:29:21.341 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:21.318+0000 7fa09e36f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:29:21.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:21.404+0000 7fa09e36f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:29:22.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:21 smithi196 ceph-mon[25696]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:22.001 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:21.676+0000 7fa09e36f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:29:22.002 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:21.804+0000 7fa09e36f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:29:22.002 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:21.902+0000 7fa09e36f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:29:22.002 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:22 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:22.001+0000 7fa09e36f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:29:22.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:21 smithi016 ceph-mon[29517]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:22.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:21 smithi016 ceph-mon[24869]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:22.309 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:22 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:22.189+0000 7fa09e36f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:29:22.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:29:22] "GET /metrics HTTP/1.1" 200 214761 "" "Prometheus/2.33.4" 2024-04-14T15:29:22.898 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:29:23.414 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:23.261+0000 7fa09e36f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:29:23.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:23.395+0000 7fa09e36f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (3m) 3m ago 4m 15.4M - ba2b418f427c e0503939608c 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (3m) 3m ago 3m 49.6M - 8.3.5 dad864ee21e9 6cd5c62d1ec5 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (3m) 3m ago 3m 49.4M - 3.5 e1d6a67b021e e2c6c6ed6283 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283 starting - - - - 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:9283 running (10m) 3m ago 10m 456M - 17.2.0 e1d6a67b021e 8678f197809c 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (10m) 3m ago 10m 48.4M 2048M 17.2.0 e1d6a67b021e e8cfd2da2fd9 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (8m) 3m ago 8m 39.2M 2048M 17.2.0 e1d6a67b021e f933c5fa37fd 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (9m) 3m ago 9m 39.0M 2048M 17.2.0 e1d6a67b021e 8629057bb8e9 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (4m) 3m ago 4m 13.9M - 1dbe0e931976 568fff30d221 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (4m) 3m ago 4m 20.5M - 1dbe0e931976 142e5a468751 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (8m) 3m ago 8m 49.7M 4017M 17.2.0 e1d6a67b021e b812f3e0fa81 2024-04-14T15:29:23.508 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (8m) 3m ago 8m 48.9M 4017M 17.2.0 e1d6a67b021e c275d1a0eaeb 2024-04-14T15:29:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (7m) 3m ago 7m 47.7M 4017M 17.2.0 e1d6a67b021e 592ce301a74a 2024-04-14T15:29:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (7m) 3m ago 7m 47.0M 4017M 17.2.0 e1d6a67b021e 1ed8730c3fc1 2024-04-14T15:29:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (6m) 3m ago 6m 49.0M 4272M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:29:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (6m) 3m ago 6m 45.7M 4272M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:29:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (6m) 3m ago 6m 45.1M 4272M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:29:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (5m) 3m ago 5m 50.4M 4272M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:29:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (3m) 3m ago 4m 44.7M - 514e6a882f6e 371540bc5a5a 2024-04-14T15:29:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (3m) 3m ago 3m 70.3M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:29:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (3m) 3m ago 3m 70.5M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:29:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (3m) 3m ago 3m 68.5M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:29:23.509 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (3m) 3m ago 3m 71.2M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:29:23.860 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:23 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=error ts=2024-04-14T15:29:23.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.196:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:29:23.860 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:23 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:23.506Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.196:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-04-14T15:29:23.861 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:23 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:29:23.874 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:29:23.874 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:29:23.874 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T15:29:23.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:29:23.874 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:29:23.874 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T15:29:23.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:29:23.874 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:29:23.874 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T15:29:23.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:29:23.875 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": {}, 2024-04-14T15:29:23.875 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:29:23.875 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:29:23.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:29:23.875 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:29:23.875 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-14T15:29:23.875 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:29:23.875 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:29:24.027 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:23.753+0000 7fa09e36f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:29:24.028 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:23.848+0000 7fa09e36f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:29:24.182 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:29:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:29:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:29:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-04-14T15:29:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "1/23 daemons upgraded", 2024-04-14T15:29:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mgr daemons" 2024-04-14T15:29:24.183 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:29:24.357 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:24.065+0000 7fa09e36f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:29:24.357 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:24.263+0000 7fa09e36f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:29:24.540 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-04-14T15:29:24.540 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[29517]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:24.540 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[29517]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:24.540 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[29517]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:24.540 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:24.540 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3511207556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:29:24.541 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:24.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[24869]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:24.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[24869]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:24.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[24869]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:24.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:24.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3511207556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:29:24.541 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:24 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:24.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:24.568+0000 7fa09e36f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:29:24.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:24 smithi196 ceph-mon[25696]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:24.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:24 smithi196 ceph-mon[25696]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:24.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:24 smithi196 ceph-mon[25696]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:24.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:24 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:24.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:24 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/3511207556' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:29:24.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:24 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:24.914 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:24.845+0000 7fa09e36f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:29:25.340 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:25 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:25.098+0000 7fa09e36f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:29:25.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:25 smithi196 ceph-mon[25696]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:25.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:25 smithi196 ceph-mon[25696]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:25.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:25 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2079965203' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:29:25.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:25 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:25.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:25 smithi016 ceph-mon[24869]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:25.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:25 smithi016 ceph-mon[24869]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:25.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:25 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2079965203' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:29:25.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:25 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:25.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:25 smithi016 ceph-mon[29517]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:25.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:25 smithi016 ceph-mon[29517]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:25.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:25 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2079965203' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:29:25.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:25 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:26.375 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:26.037+0000 7fa09e36f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:29:26.630 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:26 smithi196 ceph-mon[25696]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:26.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:26 smithi016 ceph-mon[24869]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:26.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:26 smithi016 ceph-mon[29517]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:26.917 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:26.629+0000 7fa09e36f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:29:26.917 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:26.731+0000 7fa09e36f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:29:27.382 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:26.956+0000 7fa09e36f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:29:27.382 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:26 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:29:26] ENGINE Bus STARTING 2024-04-14T15:29:27.382 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:27 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: CherryPy Checker: 2024-04-14T15:29:27.382 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:27 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: The Application mounted at '' has an empty config. 2024-04-14T15:29:27.383 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:27 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:29:27.383 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:27 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:29:27] ENGINE Serving on http://:::9283 2024-04-14T15:29:27.383 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:27 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:29:27] ENGINE Bus STARTED 2024-04-14T15:29:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:27 smithi196 ceph-mon[25696]: Standby manager daemon x restarted 2024-04-14T15:29:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:27 smithi196 ceph-mon[25696]: Standby manager daemon x started 2024-04-14T15:29:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:27 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:29:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:27 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:29:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:27 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:29:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:27 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:29:27.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[29517]: Standby manager daemon x restarted 2024-04-14T15:29:27.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[29517]: Standby manager daemon x started 2024-04-14T15:29:27.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:29:27.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:29:27.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:29:27.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:29:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[24869]: Standby manager daemon x restarted 2024-04-14T15:29:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[24869]: Standby manager daemon x started 2024-04-14T15:29:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:29:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:29:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:29:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:27 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:29:28.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:28.188Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs" 2024-04-14T15:29:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[29517]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[29517]: mgrmap e22: y(active, since 5m), standbys: x 2024-04-14T15:29:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:29:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-14T15:29:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[29517]: osdmap e96: 8 total, 8 up, 8 in 2024-04-14T15:29:28.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[24869]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:28.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[24869]: mgrmap e22: y(active, since 5m), standbys: x 2024-04-14T15:29:28.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:28.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:28.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:29:28.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-14T15:29:28.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:28 smithi016 ceph-mon[24869]: osdmap e96: 8 total, 8 up, 8 in 2024-04-14T15:29:28.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:28 smithi196 ceph-mon[25696]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:28.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:28 smithi196 ceph-mon[25696]: mgrmap e22: y(active, since 5m), standbys: x 2024-04-14T15:29:28.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:28 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:28.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:28 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' 2024-04-14T15:29:28.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:28 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:29:28.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:28 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-14T15:29:28.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:28 smithi196 ceph-mon[25696]: osdmap e96: 8 total, 8 up, 8 in 2024-04-14T15:29:29.495 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:29:29] ENGINE Bus STOPPING 2024-04-14T15:29:29.495 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:29:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:29:29.496 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:29:29] ENGINE Bus STOPPED 2024-04-14T15:29:29.496 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:29:29] ENGINE Bus STARTING 2024-04-14T15:29:29.496 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:29:29] ENGINE Serving on http://:::9283 2024-04-14T15:29:29.496 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:29:29] ENGINE Bus STARTED 2024-04-14T15:29:29.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-14T15:29:29.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: Failing over to other MGR 2024-04-14T15:29:29.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: Standby manager daemon y started 2024-04-14T15:29:29.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-14T15:29:29.496 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: mgrmap e23: x(active, starting, since 0.996972s), standbys: y 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:29:29.497 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:29:29.498 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:29:29.498 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: Manager daemon x is now available 2024-04-14T15:29:29.498 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:29.498 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:29.498 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:29.498 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:29:29.498 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:29:29.498 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:29 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:29:29.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ignoring --setuser ceph since I am not root 2024-04-14T15:29:29.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ignoring --setgroup ceph since I am not root 2024-04-14T15:29:29.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mgr[25103]: -- 172.21.15.16:0/1352741312 <== mon.1 v2:172.21.15.16:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561755858340 con 0x561755830800 2024-04-14T15:29:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-14T15:29:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: Failing over to other MGR 2024-04-14T15:29:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: Standby manager daemon y started 2024-04-14T15:29:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-14T15:29:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: mgrmap e23: x(active, starting, since 0.996972s), standbys: y 2024-04-14T15:29:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:29:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:29:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: Manager daemon x is now available 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: Failing over to other MGR 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: Standby manager daemon y started 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.24349 172.21.15.16:0/3598940478' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: mgrmap e23: x(active, starting, since 0.996972s), standbys: y 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:29:29.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: Manager daemon x is now available 2024-04-14T15:29:29.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:29.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:29.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:29.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:29:29.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:29:29.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:29:30.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:29.777+0000 7fd742c0d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:29:30.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:29 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:29.871+0000 7fd742c0d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:29:30.587 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:30.495+0000 7fd742c0d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:29:30.588 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:30.320Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.16:8443//api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-04-14T15:29:30.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[24869]: Queued rgw.foo for migration 2024-04-14T15:29:30.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[24869]: Queued rgw.smpl for migration 2024-04-14T15:29:30.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[24869]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-14T15:29:30.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[24869]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-14T15:29:30.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T15:29:30.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T15:29:30.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[24869]: mgrmap e24: x(active, since 1.99805s), standbys: y 2024-04-14T15:29:30.588 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:30.589 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:30.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[29517]: Queued rgw.foo for migration 2024-04-14T15:29:30.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[29517]: Queued rgw.smpl for migration 2024-04-14T15:29:30.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[29517]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-14T15:29:30.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[29517]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-14T15:29:30.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T15:29:30.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T15:29:30.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[29517]: mgrmap e24: x(active, since 1.99805s), standbys: y 2024-04-14T15:29:30.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:30.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:30.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:30 smithi196 ceph-mon[25696]: Queued rgw.foo for migration 2024-04-14T15:29:30.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:30 smithi196 ceph-mon[25696]: Queued rgw.smpl for migration 2024-04-14T15:29:30.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:30 smithi196 ceph-mon[25696]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-14T15:29:30.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:30 smithi196 ceph-mon[25696]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-14T15:29:30.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:30 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T15:29:30.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:30 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T15:29:30.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:30 smithi196 ceph-mon[25696]: mgrmap e24: x(active, since 1.99805s), standbys: y 2024-04-14T15:29:30.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:30 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:30.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:30 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:30.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:30.587+0000 7fd742c0d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:29:30.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:30.673+0000 7fd742c0d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:29:30.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:30.825+0000 7fd742c0d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:29:31.276 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:30.908+0000 7fd742c0d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:29:31.276 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:31.021+0000 7fd742c0d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:29:31.276 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:31.107+0000 7fd742c0d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:29:31.276 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:31.275+0000 7fd742c0d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:29:31.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:31.365+0000 7fd742c0d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:29:31.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:31 smithi016 ceph-mon[24869]: Deploying cephadm binary to smithi016 2024-04-14T15:29:31.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:31 smithi016 ceph-mon[24869]: Deploying cephadm binary to smithi196 2024-04-14T15:29:31.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:31 smithi016 ceph-mon[24869]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:31.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:31 smithi016 ceph-mon[29517]: Deploying cephadm binary to smithi016 2024-04-14T15:29:31.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:31 smithi016 ceph-mon[29517]: Deploying cephadm binary to smithi196 2024-04-14T15:29:31.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:31 smithi016 ceph-mon[29517]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:31.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:31 smithi196 ceph-mon[25696]: Deploying cephadm binary to smithi016 2024-04-14T15:29:31.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:31 smithi196 ceph-mon[25696]: Deploying cephadm binary to smithi196 2024-04-14T15:29:31.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:31 smithi196 ceph-mon[25696]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:32.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:32 smithi196 ceph-mon[25696]: [14/Apr/2024:15:29:30] ENGINE Bus STARTING 2024-04-14T15:29:32.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:32 smithi196 ceph-mon[25696]: [14/Apr/2024:15:29:30] ENGINE Serving on https://172.21.15.196:7150 2024-04-14T15:29:32.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:32 smithi196 ceph-mon[25696]: [14/Apr/2024:15:29:31] ENGINE Serving on http://172.21.15.196:8765 2024-04-14T15:29:32.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:32 smithi196 ceph-mon[25696]: [14/Apr/2024:15:29:31] ENGINE Bus STARTED 2024-04-14T15:29:32.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:32 smithi196 ceph-mon[25696]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:32.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:32 smithi196 ceph-mon[25696]: mgrmap e25: x(active, since 3s), standbys: y 2024-04-14T15:29:32.686 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:32.430+0000 7fd742c0d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:29:32.689 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:32.564+0000 7fd742c0d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:29:32.690 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[24869]: [14/Apr/2024:15:29:30] ENGINE Bus STARTING 2024-04-14T15:29:32.690 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[24869]: [14/Apr/2024:15:29:30] ENGINE Serving on https://172.21.15.196:7150 2024-04-14T15:29:32.690 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[24869]: [14/Apr/2024:15:29:31] ENGINE Serving on http://172.21.15.196:8765 2024-04-14T15:29:32.690 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[24869]: [14/Apr/2024:15:29:31] ENGINE Bus STARTED 2024-04-14T15:29:32.690 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[24869]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:32.690 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[24869]: mgrmap e25: x(active, since 3s), standbys: y 2024-04-14T15:29:32.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[29517]: [14/Apr/2024:15:29:30] ENGINE Bus STARTING 2024-04-14T15:29:32.690 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[29517]: [14/Apr/2024:15:29:30] ENGINE Serving on https://172.21.15.196:7150 2024-04-14T15:29:32.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[29517]: [14/Apr/2024:15:29:31] ENGINE Serving on http://172.21.15.196:8765 2024-04-14T15:29:32.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[29517]: [14/Apr/2024:15:29:31] ENGINE Bus STARTED 2024-04-14T15:29:32.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[29517]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:32.691 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-mon[29517]: mgrmap e25: x(active, since 3s), standbys: y 2024-04-14T15:29:32.993 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:32.763+0000 7fd742c0d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:29:32.993 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:32.853+0000 7fd742c0d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:29:33.280 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:32 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:32.992+0000 7fd742c0d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:29:33.559 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:33.324+0000 7fd742c0d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:29:33.560 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=error ts=2024-04-14T15:29:33.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.16:8443//api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs" 2024-04-14T15:29:33.560 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:33.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.16:8443//api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-04-14T15:29:33.560 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:33.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs" 2024-04-14T15:29:33.830 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:33.559+0000 7fd742c0d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:29:33.830 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:33.648+0000 7fd742c0d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:29:33.830 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:33.830+0000 7fd742c0d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:29:34.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:34 smithi196 ceph-mon[25696]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:34.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:34 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:34.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:34 smithi196 ceph-mon[25696]: mgrmap e26: x(active, since 5s), standbys: y 2024-04-14T15:29:34.793 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:34 smithi016 ceph-mon[29517]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:34.793 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:34 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:34.793 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:34 smithi016 ceph-mon[29517]: mgrmap e26: x(active, since 5s), standbys: y 2024-04-14T15:29:34.794 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:34 smithi016 ceph-mon[24869]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:34.794 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:34 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:34.794 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:34 smithi016 ceph-mon[24869]: mgrmap e26: x(active, since 5s), standbys: y 2024-04-14T15:29:35.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:34 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:34.792+0000 7fd742c0d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:29:35.382 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:35.117+0000 7fd742c0d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:29:35.382 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:35.221+0000 7fd742c0d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:29:35.656 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[29517]: Standby manager daemon y restarted 2024-04-14T15:29:35.657 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[29517]: Standby manager daemon y started 2024-04-14T15:29:35.657 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T15:29:35.657 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:29:35.657 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T15:29:35.657 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:29:35.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[24869]: Standby manager daemon y restarted 2024-04-14T15:29:35.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[24869]: Standby manager daemon y started 2024-04-14T15:29:35.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T15:29:35.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:29:35.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T15:29:35.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:29:35.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:35.381+0000 7fd742c0d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:29:35.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:29:35] ENGINE Bus STARTING 2024-04-14T15:29:35.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: CherryPy Checker: 2024-04-14T15:29:35.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: The Application mounted at '' has an empty config. 2024-04-14T15:29:35.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: 2024-04-14T15:29:35.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:29:35] ENGINE Serving on http://:::9283 2024-04-14T15:29:35.658 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: [14/Apr/2024:15:29:35] ENGINE Bus STARTED 2024-04-14T15:29:35.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:35 smithi196 ceph-mon[25696]: Standby manager daemon y restarted 2024-04-14T15:29:35.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:35 smithi196 ceph-mon[25696]: Standby manager daemon y started 2024-04-14T15:29:35.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:35 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T15:29:35.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:35 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:29:35.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:35 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T15:29:35.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:35 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.16:0/2983029441' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:29:36.110 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:35.657Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:29:36.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:36 smithi196 ceph-mon[25696]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:36.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:36 smithi196 ceph-mon[25696]: mgrmap e27: x(active, since 7s), standbys: y 2024-04-14T15:29:36.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:36 smithi016 ceph-mon[29517]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:36.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:36 smithi016 ceph-mon[29517]: mgrmap e27: x(active, since 7s), standbys: y 2024-04-14T15:29:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:36 smithi016 ceph-mon[24869]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:29:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:36 smithi016 ceph-mon[24869]: mgrmap e27: x(active, since 7s), standbys: y 2024-04-14T15:29:38.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:38 smithi196 ceph-mon[25696]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T15:29:38.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:38 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:38.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:38 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:38.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:38 smithi016 ceph-mon[29517]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T15:29:38.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:38 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:38.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:38 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:38.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:38 smithi016 ceph-mon[24869]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T15:29:38.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:38 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:38.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:38 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:39.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:39 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:29:39] "GET /metrics HTTP/1.1" 200 35351 "" "Prometheus/2.33.4" 2024-04-14T15:29:40.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: Adjusting osd_memory_target on smithi016 to 2737M 2024-04-14T15:29:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T15:29:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: Adjusting osd_memory_target on smithi016 to 2737M 2024-04-14T15:29:40.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:40 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: Adjusting osd_memory_target on smithi016 to 2737M 2024-04-14T15:29:40.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:40 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:42.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:42 smithi016 ceph-mon[24869]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T15:29:42.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:42 smithi016 ceph-mon[29517]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T15:29:42.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:29:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:29:42.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:42 smithi196 ceph-mon[25696]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T15:29:43.860 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=error ts=2024-04-14T15:29:43.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs" 2024-04-14T15:29:43.860 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:43.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:29:43.861 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:43.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs" 2024-04-14T15:29:44.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:44 smithi016 ceph-mon[24869]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:29:44.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:44 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:44.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:44 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:29:44.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:44 smithi016 ceph-mon[29517]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:29:44.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:44 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:44.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:44 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:29:44.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:44 smithi196 ceph-mon[25696]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:29:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:44 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:44 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:29:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:46 smithi016 ceph-mon[24869]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:29:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:46 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:46 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:46.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:46 smithi016 ceph-mon[29517]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:29:46.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:46 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:46.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:46 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:46.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:46 smithi196 ceph-mon[25696]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:29:46.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:46 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:46.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:46 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:47.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:47 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:47.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:47 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:47.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:47 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:47.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:47 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:47.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:47 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:47.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:47 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:48.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:48 smithi016 ceph-mon[29517]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:29:48.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:48 smithi016 ceph-mon[24869]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:29:48.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:48 smithi196 ceph-mon[25696]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:29:49.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:49 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:29:49] "GET /metrics HTTP/1.1" 200 38142 "" "Prometheus/2.33.4" 2024-04-14T15:29:50.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:49 smithi016 ceph-mon[29517]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:49 smithi016 ceph-mon[24869]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:50.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:49 smithi196 ceph-mon[25696]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:52.415 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:29:52.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.196:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.16:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:29:52.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: Adjusting osd_memory_target on smithi196 to 2992M 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:29:52.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:29:52.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:29:52.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:52 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:29:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:29:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: Adjusting osd_memory_target on smithi196 to 2992M 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: Adjusting osd_memory_target on smithi196 to 2992M 2024-04-14T15:29:52.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:52.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:52 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.025 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:52 smithi016 systemd[1]: Stopping Ceph node-exporter.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:29:53.026 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:52 smithi016 bash[58558]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter.a" found: no such container 2024-04-14T15:29:53.026 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:52 smithi016 podman[58571]: 2024-04-14 15:29:52.789385801 +0000 UTC m=+0.113485585 container died 568fff30d221b6f3b504625492b055ce914c01f72460488b86f07a6cee67394b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:29:53.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[24869]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:29:53.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[24869]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:29:53.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[24869]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-14T15:29:53.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[24869]: Reconfiguring daemon node-exporter.a on smithi016 2024-04-14T15:29:53.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[29517]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:29:53.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[29517]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:29:53.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[29517]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-14T15:29:53.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-mon[29517]: Reconfiguring daemon node-exporter.a on smithi016 2024-04-14T15:29:53.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:53 smithi196 ceph-mon[25696]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:29:53.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:53 smithi196 ceph-mon[25696]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:29:53.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:53 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:53 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:53 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:53 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:53.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:53 smithi196 ceph-mon[25696]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-14T15:29:53.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:53 smithi196 ceph-mon[25696]: Reconfiguring daemon node-exporter.a on smithi016 2024-04-14T15:29:53.842 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=error ts=2024-04-14T15:29:53.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:29:53.843 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:53.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.16:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.16 because it doesn't contain any IP SANs" 2024-04-14T15:29:53.843 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:29:53 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=warn ts=2024-04-14T15:29:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.196:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.196 because it doesn't contain any IP SANs" 2024-04-14T15:29:53.843 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:53 smithi016 podman[58571]: 2024-04-14 15:29:53.404459705 +0000 UTC m=+0.728559485 container cleanup 568fff30d221b6f3b504625492b055ce914c01f72460488b86f07a6cee67394b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:29:53.843 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:53 smithi016 bash[58571]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a 2024-04-14T15:29:53.843 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:53 smithi016 podman[58585]: 2024-04-14 15:29:53.606343418 +0000 UTC m=+0.820758342 container remove 568fff30d221b6f3b504625492b055ce914c01f72460488b86f07a6cee67394b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:29:53.843 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:53 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-14T15:29:53.843 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:53 smithi016 bash[58604]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter.a" found: no such container 2024-04-14T15:29:53.843 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:53 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-14T15:29:53.843 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:53 smithi016 systemd[1]: Stopped Ceph node-exporter.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:29:53.843 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:53 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.a.service: Consumed 2.691s CPU time. 2024-04-14T15:29:53.844 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:53 smithi016 systemd[1]: Starting Ceph node-exporter.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:29:54.109 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:53 smithi016 podman[58674]: 2024-04-14 15:29:53.745557504 +0000 UTC m=+0.017625123 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-14T15:29:54.109 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:53 smithi016 podman[58674]: 2024-04-14 15:29:53.865605775 +0000 UTC m=+0.137673381 container create cb8e94cfa1ed06f3afe4dc1379319dffad72cf87e31522983deb24334bd67835 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 podman[58674]: 2024-04-14 15:29:54.034823301 +0000 UTC m=+0.306890922 container init cb8e94cfa1ed06f3afe4dc1379319dffad72cf87e31522983deb24334bd67835 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 podman[58674]: 2024-04-14 15:29:54.038146875 +0000 UTC m=+0.310214490 container start cb8e94cfa1ed06f3afe4dc1379319dffad72cf87e31522983deb24334bd67835 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.041Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.041Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.041Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.041Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=arp 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-14T15:29:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=edac 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-14T15:29:54.111 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-14T15:29:54.112 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-14T15:29:54.112 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-14T15:29:54.112 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-14T15:29:54.112 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-14T15:29:54.112 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-14T15:29:54.112 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-14T15:29:54.112 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=os 2024-04-14T15:29:54.112 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-14T15:29:54.112 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-14T15:29:54.112 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=stat 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=time 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=uname 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-14T15:29:54.113 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.042Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-14T15:29:54.114 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a[58689]: ts=2024-04-14T15:29:54.043Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-14T15:29:54.114 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 bash[58674]: cb8e94cfa1ed06f3afe4dc1379319dffad72cf87e31522983deb24334bd67835 2024-04-14T15:29:54.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-mon[29517]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:54.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:54.361 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:29:54 smithi016 systemd[1]: Started Ceph node-exporter.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:29:54.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-mon[24869]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:54.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:54 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:54.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:54 smithi196 ceph-mon[25696]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:54.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:54 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:29:54.935 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:29:55.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:55 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:55.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:55 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:55.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:55 smithi016 ceph-mon[24869]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T15:29:55.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:55 smithi016 ceph-mon[24869]: Reconfiguring daemon alertmanager.a on smithi016 2024-04-14T15:29:55.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:55 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:55.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:55 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:55.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:55 smithi016 ceph-mon[29517]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T15:29:55.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:55 smithi016 ceph-mon[29517]: Reconfiguring daemon alertmanager.a on smithi016 2024-04-14T15:29:55.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:55 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:55.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:55 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:29:55.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:55 smithi196 ceph-mon[25696]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T15:29:55.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:55 smithi196 ceph-mon[25696]: Reconfiguring daemon alertmanager.a on smithi016 2024-04-14T15:29:55.621 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:29:55.621 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (4m) 17s ago 4m 26.6M - 0.23.0 ba2b418f427c e0503939608c 2024-04-14T15:29:55.621 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (4m) 10s ago 4m 57.4M - 8.3.5 dad864ee21e9 6cd5c62d1ec5 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (3m) 17s ago 3m 53.5M - 3.5 e1d6a67b021e e2c6c6ed6283 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283 running (36s) 10s ago 9m 501M - 19.0.0-2778-ga971fd5f b96c509eef84 ef0a2cd931a5 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:9283 running (10m) 17s ago 10m 384M - 17.2.0 e1d6a67b021e 8678f197809c 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (10m) 17s ago 10m 55.2M 2048M 17.2.0 e1d6a67b021e e8cfd2da2fd9 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (9m) 10s ago 9m 42.2M 2048M 17.2.0 e1d6a67b021e f933c5fa37fd 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (9m) 17s ago 9m 42.4M 2048M 17.2.0 e1d6a67b021e 8629057bb8e9 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 starting - - - - 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (5m) 10s ago 5m 18.6M - 1.3.1 1dbe0e931976 142e5a468751 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (8m) 17s ago 8m 52.6M 2737M 17.2.0 e1d6a67b021e b812f3e0fa81 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (8m) 17s ago 8m 52.1M 2737M 17.2.0 e1d6a67b021e c275d1a0eaeb 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (8m) 17s ago 8m 50.1M 2737M 17.2.0 e1d6a67b021e 592ce301a74a 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (7m) 17s ago 7m 49.5M 2737M 17.2.0 e1d6a67b021e 1ed8730c3fc1 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (7m) 10s ago 7m 52.3M 2992M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:29:55.622 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (7m) 10s ago 7m 49.3M 2992M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:29:55.623 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (6m) 10s ago 6m 48.5M 2992M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:29:55.623 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (6m) 10s ago 6m 52.5M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:29:55.623 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (4m) 10s ago 5m 66.3M - 2.33.4 514e6a882f6e 371540bc5a5a 2024-04-14T15:29:55.623 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (3m) 17s ago 3m 72.4M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:29:55.623 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (3m) 10s ago 3m 72.9M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:29:55.623 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (3m) 17s ago 3m 70.7M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:29:55.623 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (3m) 10s ago 3m 73.3M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:29:56.045 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:29:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:29:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T15:29:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:29:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:29:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T15:29:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T15:29:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": {}, 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:29:56.046 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:29:56.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:56 smithi016 ceph-mon[29517]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:56.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:56 smithi016 ceph-mon[29517]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:56.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:56 smithi016 ceph-mon[29517]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:56.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:56 smithi016 ceph-mon[29517]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:56.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:56 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/811617371' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:29:56.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:56 smithi016 ceph-mon[24869]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:56.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:56 smithi016 ceph-mon[24869]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:56.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:56 smithi016 ceph-mon[24869]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:56.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:56 smithi016 ceph-mon[24869]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:56.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:56 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/811617371' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:29:56.398 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:29:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:29:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:29:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:29:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-04-14T15:29:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "1/23 daemons upgraded", 2024-04-14T15:29:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "", 2024-04-14T15:29:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:29:56.399 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:29:56.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:56 smithi196 ceph-mon[25696]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:56.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:56 smithi196 ceph-mon[25696]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:29:56.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:56 smithi196 ceph-mon[25696]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:56.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:56 smithi196 ceph-mon[25696]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:56.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:56 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/811617371' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:29:56.821 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-04-14T15:29:57.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:57 smithi196 ceph-mon[25696]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:57.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:57 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2772102162' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:29:57.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:57 smithi016 ceph-mon[29517]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:57.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:57 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2772102162' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:29:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:57 smithi016 ceph-mon[24869]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:29:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:57 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2772102162' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:29:58.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:58 smithi196 ceph-mon[25696]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:58.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:58 smithi016 ceph-mon[29517]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:58 smithi016 ceph-mon[24869]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:29:59.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:29:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:29:59] "GET /metrics HTTP/1.1" 200 38142 "" "Prometheus/2.33.4" 2024-04-14T15:30:00.093 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:59 smithi016 ceph-mon[29517]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:00.093 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:29:59 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:00.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:59 smithi016 ceph-mon[24869]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:00.094 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:29:59 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:00.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:59 smithi196 ceph-mon[25696]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:00.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:29:59 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:01.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:00 smithi016 ceph-mon[24869]: overall HEALTH_OK 2024-04-14T15:30:01.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:00 smithi016 ceph-mon[29517]: overall HEALTH_OK 2024-04-14T15:30:01.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:00 smithi196 ceph-mon[25696]: overall HEALTH_OK 2024-04-14T15:30:02.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:01 smithi016 ceph-mon[24869]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:02.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:01 smithi016 ceph-mon[29517]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:02.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:01 smithi196 ceph-mon[25696]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:02.564 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:30:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:30:02.666 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:02.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.196:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.16:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:30:02.851 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:02 smithi016 systemd[1]: Stopping Ceph alertmanager.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:02.851 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:02 smithi016 bash[59076]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager.a" found: no such container 2024-04-14T15:30:03.110 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[49707]: level=info ts=2024-04-14T15:30:02.851Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-14T15:30:03.111 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:02 smithi016 podman[59089]: 2024-04-14 15:30:02.872209835 +0000 UTC m=+0.131358753 container died e0503939608c0a3ebbbb1f5c36f412351974b5fcd1e237f84f23ff6ba90092b6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:03.491 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:03 smithi016 podman[59089]: 2024-04-14 15:30:03.251462482 +0000 UTC m=+0.510611397 container cleanup e0503939608c0a3ebbbb1f5c36f412351974b5fcd1e237f84f23ff6ba90092b6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:03.491 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:03 smithi016 bash[59089]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a 2024-04-14T15:30:03.491 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:03 smithi016 bash[59115]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager.a" found: no such container 2024-04-14T15:30:03.779 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:03 smithi016 podman[59103]: 2024-04-14 15:30:03.491324657 +0000 UTC m=+0.616954079 container remove e0503939608c0a3ebbbb1f5c36f412351974b5fcd1e237f84f23ff6ba90092b6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:03.779 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:03 smithi016 podman[59103]: 2024-04-14 15:30:03.562435375 +0000 UTC m=+0.688064797 volume remove acd29d35ff693085daa0538dce60b8849e3790484bcc627d5184ab31648b4179 2024-04-14T15:30:03.779 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:03 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-14T15:30:03.779 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:03 smithi016 systemd[1]: Stopped Ceph alertmanager.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:03.779 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:03 smithi016 systemd[1]: Starting Ceph alertmanager.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:03.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:03.505Z caller=notifier.go:526 level=error component=notifier alertmanager=http://172.21.15.16:9093/api/v2/alerts count=3 msg="Error sending alert" err="Post \"http://172.21.15.16:9093/api/v2/alerts\": dial tcp 172.21.15.16:9093: connect: connection refused" 2024-04-14T15:30:04.111 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:03 smithi016 podman[59187]: 2024-04-14 15:30:03.779555787 +0000 UTC m=+0.092449106 volume create a68c846189e240045ee5fc2ecbaffbc306b0d084f33f5745d9fff0e4949e145f 2024-04-14T15:30:04.111 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:03 smithi016 podman[59187]: 2024-04-14 15:30:03.706276521 +0000 UTC m=+0.019169862 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-14T15:30:04.111 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:03 smithi016 podman[59187]: 2024-04-14 15:30:03.88023819 +0000 UTC m=+0.193131510 container create a29a63fa714603b4bde3bda643d2e3f72b15b8df08cb939f4ce3a81c1b65d4a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:04.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-mon[29517]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:04.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-mon[29517]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-mon[24869]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-mon[24869]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 podman[59187]: 2024-04-14 15:30:04.350300132 +0000 UTC m=+0.663193463 container init a29a63fa714603b4bde3bda643d2e3f72b15b8df08cb939f4ce3a81c1b65d4a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 podman[59187]: 2024-04-14 15:30:04.353302304 +0000 UTC m=+0.666195627 container start a29a63fa714603b4bde3bda643d2e3f72b15b8df08cb939f4ce3a81c1b65d4a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=info ts=2024-04-14T15:30:04.364Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-14T15:30:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=info ts=2024-04-14T15:30:04.364Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-14T15:30:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=info ts=2024-04-14T15:30:04.366Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.16 port=9094 2024-04-14T15:30:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=info ts=2024-04-14T15:30:04.367Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-14T15:30:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=info ts=2024-04-14T15:30:04.393Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T15:30:04.612 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=info ts=2024-04-14T15:30:04.393Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T15:30:04.612 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=info ts=2024-04-14T15:30:04.395Z caller=main.go:518 msg=Listening address=:9093 2024-04-14T15:30:04.612 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=info ts=2024-04-14T15:30:04.395Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-14T15:30:04.612 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 bash[59187]: a29a63fa714603b4bde3bda643d2e3f72b15b8df08cb939f4ce3a81c1b65d4a0 2024-04-14T15:30:04.612 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:04 smithi016 systemd[1]: Started Ceph alertmanager.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:04.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:04 smithi196 ceph-mon[25696]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:04.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:04 smithi196 ceph-mon[25696]: from='client.14841 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:05.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:05.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:05.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[24869]: Reconfiguring iscsi.foo.smithi016.sijcrc (dependencies changed)... 2024-04-14T15:30:05.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:05.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:05.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:05.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[24869]: Reconfiguring daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:30:05.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:05.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:05.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[29517]: Reconfiguring iscsi.foo.smithi016.sijcrc (dependencies changed)... 2024-04-14T15:30:05.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:05.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:05.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:05.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:05 smithi016 ceph-mon[29517]: Reconfiguring daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:30:05.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:05 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:05 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:05 smithi196 ceph-mon[25696]: Reconfiguring iscsi.foo.smithi016.sijcrc (dependencies changed)... 2024-04-14T15:30:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:05 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:05 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:05 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:05 smithi196 ceph-mon[25696]: Reconfiguring daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:30:06.396 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:06 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=info ts=2024-04-14T15:30:06.368Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001030429s 2024-04-14T15:30:06.655 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:06 smithi016 ceph-mon[29517]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:06.655 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:06 smithi016 ceph-mon[24869]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:06.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:06 smithi196 ceph-mon[25696]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:08.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:07.948+0000 7fa023fcd700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:08.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Creating ceph-iscsi config... 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Traceback (most recent call last): 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: return _run_code(code, main_globals, None, 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: exec(code, run_globals) 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:08.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Traceback (most recent call last): 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: return self.event_loop.get_result(coro, timeout) 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T15:30:08.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: return future.result(timeout) 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: return self.__get_result() 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: raise self._exception 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: config_blobs=daemon_spec.final_config, 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Creating ceph-iscsi config... 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:08.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Traceback (most recent call last): 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: return _run_code(code, main_globals, None, 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: exec(code, run_globals) 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:08.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:08.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:08 smithi016 ceph-mon[29517]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:08.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:08 smithi016 ceph-mon[24869]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:08.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:08 smithi196 ceph-mon[25696]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:09.666 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:30:09] "GET /metrics HTTP/1.1" 200 38145 "" "Prometheus/2.33.4" 2024-04-14T15:30:09.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:09 smithi196 ceph-mon[25696]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:09.667 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:09.667 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:09.667 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:09.667 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:09.667 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.667 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.667 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:09.668 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: return future.result(timeout) 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T15:30:09.669 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.__get_result() 2024-04-14T15:30:09.670 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T15:30:09.670 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise self._exception 2024-04-14T15:30:09.670 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T15:30:09.670 INFO:journalctl@ceph.mon.b.smithi196.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T15:30:09.670 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T15:30:09.670 INFO:journalctl@ceph.mon.b.smithi196.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T15:30:09.670 INFO:journalctl@ceph.mon.b.smithi196.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:09.670 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:09.670 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:09.670 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:09.670 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:09.671 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:09.672 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:09.672 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:09.672 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.672 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.672 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:09 smithi196 ceph-mon[25696]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:30:09.672 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:09 smithi196 ceph-mon[25696]: Reconfiguring daemon prometheus.a on smithi196 2024-04-14T15:30:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:09 smithi016 ceph-mon[29517]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: exec(code, run_globals) 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T15:30:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: return future.result(timeout) 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: return self.__get_result() 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: raise self._exception 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:09.863 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: exec(code, run_globals) 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:09.864 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:09.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:09 smithi016 ceph-mon[24869]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:09.865 INFO:journalctl@ceph.mon.a.smithi016.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:09.865 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: exec(code, run_globals) 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:09.866 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T15:30:09.867 INFO:journalctl@ceph.mon.a.smithi016.stdout: return future.result(timeout) 2024-04-14T15:30:09.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T15:30:09.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.__get_result() 2024-04-14T15:30:09.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T15:30:09.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: raise self._exception 2024-04-14T15:30:09.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T15:30:09.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T15:30:09.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T15:30:09.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T15:30:09.868 INFO:journalctl@ceph.mon.a.smithi016.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: exec(code, run_globals) 2024-04-14T15:30:09.869 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:09.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:09.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:09.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:09.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:09.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:09.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:09.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.870 INFO:journalctl@ceph.mon.a.smithi016.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.870 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:09 smithi016 ceph-mon[24869]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:30:09.870 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:09 smithi016 ceph-mon[24869]: Reconfiguring daemon prometheus.a on smithi196 2024-04-14T15:30:09.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:09.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:09.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:09.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:09.871 INFO:journalctl@ceph.mon.c.smithi016.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:09.871 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:09 smithi016 ceph-mon[29517]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:30:09.871 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:09 smithi016 ceph-mon[29517]: Reconfiguring daemon prometheus.a on smithi196 2024-04-14T15:30:10.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:10 smithi016 ceph-mon[29517]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:10.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:10 smithi016 ceph-mon[24869]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:10.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:10 smithi196 ceph-mon[25696]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:12.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:12 smithi016 ceph-mon[24869]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-14T15:30:12.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:12 smithi016 ceph-mon[29517]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-14T15:30:12.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:12 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[25099]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:30:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:30:12.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:12 smithi196 ceph-mon[25696]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-14T15:30:12.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:12.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.196:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.16:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:30:14.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:14 smithi016 ceph-mon[24869]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-14T15:30:14.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:14 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:14.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:14 smithi016 ceph-mon[29517]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-14T15:30:14.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:14 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:14.863 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:30:14 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=info ts=2024-04-14T15:30:14.370Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003488412s 2024-04-14T15:30:14.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:14 smithi196 ceph-mon[25696]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-14T15:30:14.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:14 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:16.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:16 smithi016 ceph-mon[29517]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-14T15:30:16.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:16 smithi016 ceph-mon[24869]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-14T15:30:16.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:16 smithi196 ceph-mon[25696]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-14T15:30:18.615 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:18 smithi016 ceph-mon[29517]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T15:30:18.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:18 smithi016 ceph-mon[24869]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T15:30:18.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-mon[25696]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T15:30:18.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 systemd[1]: Stopping Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:18.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 bash[50396]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T15:30:18.975 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.698Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T15:30:18.975 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.699Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-14T15:30:18.975 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.699Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-14T15:30:18.975 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.699Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-14T15:30:18.975 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.699Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-14T15:30:18.975 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.699Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-14T15:30:18.975 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.699Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-14T15:30:18.975 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.699Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T15:30:18.976 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.699Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T15:30:18.976 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.699Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T15:30:18.976 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.699Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-14T15:30:18.976 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[42635]: ts=2024-04-14T15:30:18.699Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-14T15:30:18.976 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:18 smithi196 podman[50409]: 2024-04-14 15:30:18.70905236 +0000 UTC m=+0.159890683 container died 371540bc5a5a30e30777ad3e62399997be823ba79ee232ff1d457200bb3fbd9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:19.349 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 podman[50409]: 2024-04-14 15:30:19.081819742 +0000 UTC m=+0.532658061 container cleanup 371540bc5a5a30e30777ad3e62399997be823ba79ee232ff1d457200bb3fbd9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:19.349 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 bash[50409]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T15:30:19.350 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 bash[50435]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T15:30:19.608 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 podman[50423]: 2024-04-14 15:30:19.34965462 +0000 UTC m=+0.638476931 container remove 371540bc5a5a30e30777ad3e62399997be823ba79ee232ff1d457200bb3fbd9f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:19.608 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-14T15:30:19.608 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 systemd[1]: Stopped Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:19.609 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@prometheus.a.service: Consumed 1.493s CPU time. 2024-04-14T15:30:19.609 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 systemd[1]: Starting Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:19.609 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 podman[50498]: 2024-04-14 15:30:19.488822102 +0000 UTC m=+0.019405968 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-14T15:30:19.609 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 podman[50498]: 2024-04-14 15:30:19.608992761 +0000 UTC m=+0.139576618 container create c4255d1340850c0eebd5a88f911ac040da4e6c088f8be10dfd2cd19416b5a052 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:19.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-mon[25696]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T15:30:19.894 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 podman[50498]: 2024-04-14 15:30:19.828078798 +0000 UTC m=+0.358662656 container init c4255d1340850c0eebd5a88f911ac040da4e6c088f8be10dfd2cd19416b5a052 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:19.894 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 podman[50498]: 2024-04-14 15:30:19.830680284 +0000 UTC m=+0.361264142 container start c4255d1340850c0eebd5a88f911ac040da4e6c088f8be10dfd2cd19416b5a052 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:30:19.894 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.853Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-14T15:30:19.894 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.853Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-14T15:30:19.894 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.853Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-14T15:30:19.895 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.853Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi196 (none))" 2024-04-14T15:30:19.895 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.854Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T15:30:19.895 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.854Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T15:30:19.895 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.855Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T15:30:19.895 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.856Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-14T15:30:19.895 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.856Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-14T15:30:19.895 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.858Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T15:30:19.895 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.858Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.022µs 2024-04-14T15:30:19.895 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:19.858Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T15:30:19.895 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 bash[50498]: c4255d1340850c0eebd5a88f911ac040da4e6c088f8be10dfd2cd19416b5a052 2024-04-14T15:30:20.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:19 smithi016 ceph-mon[24869]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T15:30:20.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:19 smithi016 ceph-mon[29517]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T15:30:20.164 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:19 smithi196 systemd[1]: Started Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:20.164 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:20.065Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-04-14T15:30:20.516 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:20 smithi196 systemd[1]: Stopping Ceph node-exporter.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:20.517 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:20.325Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-04-14T15:30:20.517 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:20.326Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-04-14T15:30:20.517 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:20.326Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=37.348µs wal_replay_duration=468.062664ms total_replay_duration=468.120039ms 2024-04-14T15:30:20.517 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:20.331Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T15:30:20.517 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:20.331Z caller=main.go:947 level=info msg="TSDB started" 2024-04-14T15:30:20.517 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:20.331Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T15:30:20.517 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:20.362Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=31.254619ms db_storage=981ns remote_storage=1.649µs web_handler=412ns query_engine=781ns scrape=536.672µs scrape_sd=173.683µs notify=20.758µs notify_sd=20.939µs rules=29.963365ms 2024-04-14T15:30:20.517 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:20.362Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-14T15:30:20.768 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:20 smithi196 bash[50633]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter.b" found: no such container 2024-04-14T15:30:20.768 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:20 smithi196 podman[50646]: 2024-04-14 15:30:20.634003324 +0000 UTC m=+0.108448923 container died 142e5a468751044dade65065274010ce8ae8b745f4d909cdac85b8ddc7e25aa2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:30:21.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2892367202' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:30:21.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:21.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[29517]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-14T15:30:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[29517]: Reconfiguring daemon node-exporter.b on smithi196 2024-04-14T15:30:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4108933547' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3043941881"}]: dispatch 2024-04-14T15:30:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2892367202' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:30:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[24869]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-14T15:30:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[24869]: Reconfiguring daemon node-exporter.b on smithi196 2024-04-14T15:30:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:20 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4108933547' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3043941881"}]: dispatch 2024-04-14T15:30:21.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2892367202' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:30:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-mon[25696]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-14T15:30:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-mon[25696]: Reconfiguring daemon node-exporter.b on smithi196 2024-04-14T15:30:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:20 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/4108933547' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3043941881"}]: dispatch 2024-04-14T15:30:21.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:20 smithi196 podman[50646]: 2024-04-14 15:30:20.973711124 +0000 UTC m=+0.448156709 container cleanup 142e5a468751044dade65065274010ce8ae8b745f4d909cdac85b8ddc7e25aa2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:30:21.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:20 smithi196 bash[50646]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b 2024-04-14T15:30:21.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:20 smithi196 bash[50674]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter.b" found: no such container 2024-04-14T15:30:21.500 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 podman[50660]: 2024-04-14 15:30:21.275318082 +0000 UTC m=+0.645675151 container remove 142e5a468751044dade65065274010ce8ae8b745f4d909cdac85b8ddc7e25aa2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:30:21.500 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-14T15:30:21.500 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-14T15:30:21.500 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 systemd[1]: Stopped Ceph node-exporter.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:21.501 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.b.service: Consumed 2.897s CPU time. 2024-04-14T15:30:21.501 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 systemd[1]: Starting Ceph node-exporter.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:21.763 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 podman[50746]: 2024-04-14 15:30:21.40310037 +0000 UTC m=+0.017077057 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-14T15:30:21.763 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 podman[50746]: 2024-04-14 15:30:21.562120101 +0000 UTC m=+0.176096832 container create bea8d0b8e46e0751891ed3f036e1e23b2f7660cce66d36cf6311596e97f87a0a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:30:22.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/4108933547' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3043941881"}]': finished 2024-04-14T15:30:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-mon[25696]: osdmap e97: 8 total, 8 up, 8 in 2024-04-14T15:30:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-mon[25696]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-04-14T15:30:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/4151065295' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2330461653"}]: dispatch 2024-04-14T15:30:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2330461653"}]: dispatch 2024-04-14T15:30:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:22.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:22.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 podman[50746]: 2024-04-14 15:30:21.764184147 +0000 UTC m=+0.378160834 container init bea8d0b8e46e0751891ed3f036e1e23b2f7660cce66d36cf6311596e97f87a0a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:30:22.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 podman[50746]: 2024-04-14 15:30:21.766856973 +0000 UTC m=+0.380833671 container start bea8d0b8e46e0751891ed3f036e1e23b2f7660cce66d36cf6311596e97f87a0a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.770Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.770Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.770Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.770Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=arp 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-14T15:30:22.166 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-14T15:30:22.167 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=edac 2024-04-14T15:30:22.167 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-14T15:30:22.167 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-14T15:30:22.167 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-14T15:30:22.167 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-14T15:30:22.167 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-14T15:30:22.167 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-14T15:30:22.167 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=os 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-14T15:30:22.168 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=stat 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=time 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=uname 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.771Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b[50760]: ts=2024-04-14T15:30:21.772Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 bash[50746]: bea8d0b8e46e0751891ed3f036e1e23b2f7660cce66d36cf6311596e97f87a0a 2024-04-14T15:30:22.169 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:30:21 smithi196 systemd[1]: Started Ceph node-exporter.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:22.253 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4108933547' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3043941881"}]': finished 2024-04-14T15:30:22.254 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[29517]: osdmap e97: 8 total, 8 up, 8 in 2024-04-14T15:30:22.254 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[29517]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-04-14T15:30:22.254 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4151065295' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2330461653"}]: dispatch 2024-04-14T15:30:22.254 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2330461653"}]: dispatch 2024-04-14T15:30:22.254 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:22.254 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:22.254 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4108933547' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/3043941881"}]': finished 2024-04-14T15:30:22.254 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[24869]: osdmap e97: 8 total, 8 up, 8 in 2024-04-14T15:30:22.254 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[24869]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-04-14T15:30:22.255 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4151065295' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2330461653"}]: dispatch 2024-04-14T15:30:22.255 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2330461653"}]: dispatch 2024-04-14T15:30:22.255 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:22.255 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:21 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:23.194 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[29517]: Reconfiguring grafana.a (dependencies changed)... 2024-04-14T15:30:23.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[29517]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-14T15:30:23.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2330461653"}]': finished 2024-04-14T15:30:23.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[29517]: osdmap e98: 8 total, 8 up, 8 in 2024-04-14T15:30:23.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:23.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:23.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:30:23.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[29517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:30:23.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[29517]: Reconfiguring daemon grafana.a on smithi196 2024-04-14T15:30:23.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/503526221' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3043941881"}]: dispatch 2024-04-14T15:30:23.195 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3043941881"}]: dispatch 2024-04-14T15:30:23.195 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[24869]: Reconfiguring grafana.a (dependencies changed)... 2024-04-14T15:30:23.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[24869]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-14T15:30:23.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2330461653"}]': finished 2024-04-14T15:30:23.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[24869]: osdmap e98: 8 total, 8 up, 8 in 2024-04-14T15:30:23.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:23.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:23.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:30:23.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[24869]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:30:23.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[24869]: Reconfiguring daemon grafana.a on smithi196 2024-04-14T15:30:23.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/503526221' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3043941881"}]: dispatch 2024-04-14T15:30:23.196 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:22 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3043941881"}]: dispatch 2024-04-14T15:30:23.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:22 smithi196 ceph-mon[25696]: Reconfiguring grafana.a (dependencies changed)... 2024-04-14T15:30:23.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:22 smithi196 ceph-mon[25696]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-14T15:30:23.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:22 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2330461653"}]': finished 2024-04-14T15:30:23.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:22 smithi196 ceph-mon[25696]: osdmap e98: 8 total, 8 up, 8 in 2024-04-14T15:30:23.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:22 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:23.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:22 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:23.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:22 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:30:23.467 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:22 smithi196 ceph-mon[25696]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T15:30:23.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:22 smithi196 ceph-mon[25696]: Reconfiguring daemon grafana.a on smithi196 2024-04-14T15:30:23.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:22 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/503526221' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3043941881"}]: dispatch 2024-04-14T15:30:23.468 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:22 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3043941881"}]: dispatch 2024-04-14T15:30:24.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:24 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3043941881"}]': finished 2024-04-14T15:30:24.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:24 smithi016 ceph-mon[29517]: osdmap e99: 8 total, 8 up, 8 in 2024-04-14T15:30:24.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:24 smithi016 ceph-mon[29517]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T15:30:24.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:24 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/729412624' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1935194990"}]: dispatch 2024-04-14T15:30:24.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:24 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1935194990"}]: dispatch 2024-04-14T15:30:24.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:24 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3043941881"}]': finished 2024-04-14T15:30:24.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:24 smithi016 ceph-mon[24869]: osdmap e99: 8 total, 8 up, 8 in 2024-04-14T15:30:24.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:24 smithi016 ceph-mon[24869]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T15:30:24.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:24 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/729412624' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1935194990"}]: dispatch 2024-04-14T15:30:24.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:24 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1935194990"}]: dispatch 2024-04-14T15:30:24.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:24 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/3043941881"}]': finished 2024-04-14T15:30:24.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:24 smithi196 ceph-mon[25696]: osdmap e99: 8 total, 8 up, 8 in 2024-04-14T15:30:24.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:24 smithi196 ceph-mon[25696]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T15:30:24.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:24 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/729412624' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1935194990"}]: dispatch 2024-04-14T15:30:24.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:24 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1935194990"}]: dispatch 2024-04-14T15:30:25.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:25 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1935194990"}]': finished 2024-04-14T15:30:25.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:25 smithi016 ceph-mon[29517]: osdmap e100: 8 total, 8 up, 8 in 2024-04-14T15:30:25.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:25 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2069060968' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/42332771"}]: dispatch 2024-04-14T15:30:25.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:25 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1935194990"}]': finished 2024-04-14T15:30:25.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:25 smithi016 ceph-mon[24869]: osdmap e100: 8 total, 8 up, 8 in 2024-04-14T15:30:25.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:25 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2069060968' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/42332771"}]: dispatch 2024-04-14T15:30:25.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:25 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/1935194990"}]': finished 2024-04-14T15:30:25.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:25 smithi196 ceph-mon[25696]: osdmap e100: 8 total, 8 up, 8 in 2024-04-14T15:30:25.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:25 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2069060968' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/42332771"}]: dispatch 2024-04-14T15:30:26.164 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:25 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:30:25] "GET /metrics HTTP/1.1" 200 38138 "" "Prometheus/2.33.4" 2024-04-14T15:30:26.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:26 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2069060968' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/42332771"}]': finished 2024-04-14T15:30:26.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:26 smithi196 ceph-mon[25696]: osdmap e101: 8 total, 8 up, 8 in 2024-04-14T15:30:26.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:26 smithi196 ceph-mon[25696]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:26.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:26 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/550432264' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2858662230"}]: dispatch 2024-04-14T15:30:26.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:26 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2069060968' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/42332771"}]': finished 2024-04-14T15:30:26.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:26 smithi016 ceph-mon[29517]: osdmap e101: 8 total, 8 up, 8 in 2024-04-14T15:30:26.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:26 smithi016 ceph-mon[29517]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:26.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:26 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/550432264' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2858662230"}]: dispatch 2024-04-14T15:30:26.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:26 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/2069060968' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/42332771"}]': finished 2024-04-14T15:30:26.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:26 smithi016 ceph-mon[24869]: osdmap e101: 8 total, 8 up, 8 in 2024-04-14T15:30:26.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:26 smithi016 ceph-mon[24869]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:26.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:26 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/550432264' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2858662230"}]: dispatch 2024-04-14T15:30:27.200 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:30:27.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:27 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/550432264' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2858662230"}]': finished 2024-04-14T15:30:27.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:27 smithi016 ceph-mon[29517]: osdmap e102: 8 total, 8 up, 8 in 2024-04-14T15:30:27.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:27 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1544059337' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/656697893"}]: dispatch 2024-04-14T15:30:27.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:27 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/656697893"}]: dispatch 2024-04-14T15:30:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:27 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/550432264' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2858662230"}]': finished 2024-04-14T15:30:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:27 smithi016 ceph-mon[24869]: osdmap e102: 8 total, 8 up, 8 in 2024-04-14T15:30:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:27 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1544059337' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/656697893"}]: dispatch 2024-04-14T15:30:27.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:27 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/656697893"}]: dispatch 2024-04-14T15:30:27.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:27 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/550432264' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2858662230"}]': finished 2024-04-14T15:30:27.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:27 smithi196 ceph-mon[25696]: osdmap e102: 8 total, 8 up, 8 in 2024-04-14T15:30:27.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:27 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1544059337' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/656697893"}]: dispatch 2024-04-14T15:30:27.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:27 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/656697893"}]: dispatch 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 starting - - - - 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 running (4m) 42s ago 4m 57.4M - 8.3.5 dad864ee21e9 6cd5c62d1ec5 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (4m) 49s ago 4m 53.5M - 3.5 e1d6a67b021e e2c6c6ed6283 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283 running (68s) 42s ago 9m 501M - 19.0.0-2778-ga971fd5f b96c509eef84 ef0a2cd931a5 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:9283 running (11m) 49s ago 11m 384M - 17.2.0 e1d6a67b021e 8678f197809c 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (11m) 49s ago 11m 55.2M 2048M 17.2.0 e1d6a67b021e e8cfd2da2fd9 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (10m) 42s ago 10m 42.2M 2048M 17.2.0 e1d6a67b021e f933c5fa37fd 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (10m) 49s ago 10m 42.4M 2048M 17.2.0 e1d6a67b021e 8629057bb8e9 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 starting - - - - 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 starting - - - - 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (9m) 49s ago 9m 52.6M 2737M 17.2.0 e1d6a67b021e b812f3e0fa81 2024-04-14T15:30:27.876 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (9m) 49s ago 9m 52.1M 2737M 17.2.0 e1d6a67b021e c275d1a0eaeb 2024-04-14T15:30:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (8m) 49s ago 8m 50.1M 2737M 17.2.0 e1d6a67b021e 592ce301a74a 2024-04-14T15:30:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (8m) 49s ago 8m 49.5M 2737M 17.2.0 e1d6a67b021e 1ed8730c3fc1 2024-04-14T15:30:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (7m) 42s ago 7m 52.3M 2992M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:30:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (7m) 42s ago 7m 49.3M 2992M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:30:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (7m) 42s ago 7m 48.5M 2992M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:30:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (6m) 42s ago 6m 52.5M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:30:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 starting - - - - 2024-04-14T15:30:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (4m) 49s ago 4m 72.4M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:30:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (4m) 42s ago 4m 72.9M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:30:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (4m) 49s ago 4m 70.7M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:30:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (4m) 42s ago 4m 73.3M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:30:28.286 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:30:28.286 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": {}, 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:30:28.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:30:28.288 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:30:28.288 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-04-14T15:30:28.288 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T15:30:28.288 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:30:28.288 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:30:28.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:28 smithi016 ceph-mon[24869]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/656697893"}]': finished 2024-04-14T15:30:28.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:28 smithi016 ceph-mon[24869]: osdmap e103: 8 total, 8 up, 8 in 2024-04-14T15:30:28.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:28 smithi016 ceph-mon[24869]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:28.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:28 smithi016 ceph-mon[24869]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:28.288 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:28 smithi016 ceph-mon[24869]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:28.289 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:28 smithi016 ceph-mon[29517]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/656697893"}]': finished 2024-04-14T15:30:28.289 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:28 smithi016 ceph-mon[29517]: osdmap e103: 8 total, 8 up, 8 in 2024-04-14T15:30:28.289 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:28 smithi016 ceph-mon[29517]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:28.289 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:28 smithi016 ceph-mon[29517]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:28.289 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:28 smithi016 ceph-mon[29517]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:28 smithi196 ceph-mon[25696]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/656697893"}]': finished 2024-04-14T15:30:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:28 smithi196 ceph-mon[25696]: osdmap e103: 8 total, 8 up, 8 in 2024-04-14T15:30:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:28 smithi196 ceph-mon[25696]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:28 smithi196 ceph-mon[25696]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:28 smithi196 ceph-mon[25696]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:28.635 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:30:28.635 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:30:28.635 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:30:28.636 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:30:28.636 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [], 2024-04-14T15:30:28.636 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "1/23 daemons upgraded", 2024-04-14T15:30:28.636 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "", 2024-04-14T15:30:28.636 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:30:28.636 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:30:28.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:28 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:30:29.049 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-04-14T15:30:29.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:29 smithi016 ceph-mon[29517]: from='client.25180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:29.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:29 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1176670533' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:30:29.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:29 smithi016 ceph-mon[29517]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:29.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:29 smithi016 ceph-mon[24869]: from='client.25180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:29.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:29 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1176670533' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:30:29.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:29 smithi016 ceph-mon[24869]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:29.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:29 smithi196 ceph-mon[25696]: from='client.25180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:29.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:29 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1176670533' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:30:29.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:29 smithi196 ceph-mon[25696]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:30:30.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:30 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/4106340140' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:30:30.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:30 smithi196 ceph-mon[25696]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-04-14T15:30:30.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:30 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:30.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:30 smithi196 ceph-mon[25696]: from='client.25114 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:30.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:30 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4106340140' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:30:30.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:30 smithi016 ceph-mon[29517]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-04-14T15:30:30.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:30 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:30.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:30 smithi016 ceph-mon[29517]: from='client.25114 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:30.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:30 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/4106340140' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:30:30.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:30 smithi016 ceph-mon[24869]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-04-14T15:30:30.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:30 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:30.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:30 smithi016 ceph-mon[24869]: from='client.25114 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:32.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:32 smithi196 ceph-mon[25696]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-14T15:30:32.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:32 smithi016 ceph-mon[29517]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-14T15:30:32.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:32 smithi016 ceph-mon[24869]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-14T15:30:33.505 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:33 smithi196 systemd[1]: Stopping Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:33.505 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:33 smithi196 bash[51038]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana.a" found: no such container 2024-04-14T15:30:33.506 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[41839]: t=2024-04-14T15:30:33+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-14T15:30:33.506 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:33 smithi196 podman[51051]: 2024-04-14 15:30:33.460490089 +0000 UTC m=+0.116468390 container died 6cd5c62d1ec5d873d24036adf57e1eaa91a2a88c296c6430b3e8cbdcb3cb1db0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.component=ubi8-container, name=ubi8, io.openshift.expose-services=, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., maintainer=Paul Cuzner , release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, architecture=x86_64) 2024-04-14T15:30:33.767 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:30:34.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:33 smithi196 podman[51051]: 2024-04-14 15:30:33.858283068 +0000 UTC m=+0.514261377 container cleanup 6cd5c62d1ec5d873d24036adf57e1eaa91a2a88c296c6430b3e8cbdcb3cb1db0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , vcs-type=git, architecture=x86_64, distribution-scope=public, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, name=ubi8, vendor=Red Hat, Inc.) 2024-04-14T15:30:34.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:33 smithi196 bash[51051]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a 2024-04-14T15:30:34.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:33 smithi196 bash[51078]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana.a" found: no such container 2024-04-14T15:30:34.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-mon[25696]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:30:34.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 podman[51065]: 2024-04-14 15:30:34.144366846 +0000 UTC m=+0.684294845 container remove 6cd5c62d1ec5d873d24036adf57e1eaa91a2a88c296c6430b3e8cbdcb3cb1db0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, version=8.5, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, distribution-scope=public, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-04-14T15:30:34.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-14T15:30:34.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:34.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Consumed 2.595s CPU time. 2024-04-14T15:30:34.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:34.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 podman[51149]: 2024-04-14 15:30:34.279834439 +0000 UTC m=+0.017007293 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:30:34.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:34 smithi016 ceph-mon[29517]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:30:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:34 smithi016 ceph-mon[24869]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:30:34.701 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 podman[51149]: 2024-04-14 15:30:34.435863231 +0000 UTC m=+0.173036118 container create 848a2eb010e78f921444581433b15eececa9e6122574e526c111e309327ae6a7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, architecture=x86_64, distribution-scope=public, vcs-type=git, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.tags=base rhel8, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-14T15:30:34.702 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 podman[51149]: 2024-04-14 15:30:34.671736017 +0000 UTC m=+0.408908869 container init 848a2eb010e78f921444581433b15eececa9e6122574e526c111e309327ae6a7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, maintainer=Paul Cuzner , name=ubi8, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., version=8.5, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762) 2024-04-14T15:30:34.702 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 podman[51149]: 2024-04-14 15:30:34.674545377 +0000 UTC m=+0.411718225 container start 848a2eb010e78f921444581433b15eececa9e6122574e526c111e309327ae6a7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, distribution-scope=public, release=236.1648460182, version=8.5, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, vcs-type=git, name=ubi8, io.openshift.tags=base rhel8, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.component=ubi8-container) 2024-04-14T15:30:34.968 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:34] ENGINE Bus STOPPING 2024-04-14T15:30:34.969 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:30:34.969 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:34] ENGINE Bus STOPPED 2024-04-14T15:30:34.969 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:34] ENGINE Bus STARTING 2024-04-14T15:30:34.969 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:34] ENGINE Serving on http://:::9283 2024-04-14T15:30:34.969 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:34] ENGINE Bus STARTED 2024-04-14T15:30:34.969 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[51164]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:30:34.969 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[51164]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:30:34.969 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 bash[51149]: 848a2eb010e78f921444581433b15eececa9e6122574e526c111e309327ae6a7 2024-04-14T15:30:34.969 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:34.969 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:34 smithi196 podman[51180]: 2024-04-14 15:30:34.783933139 +0000 UTC m=+0.075174510 container died 848a2eb010e78f921444581433b15eececa9e6122574e526c111e309327ae6a7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vendor=Red Hat, Inc., version=8.5, architecture=x86_64, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, name=ubi8, vcs-type=git, distribution-scope=public) 2024-04-14T15:30:35.367 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:34] ENGINE Bus STOPPING 2024-04-14T15:30:35.367 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:30:35.367 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:34] ENGINE Bus STOPPED 2024-04-14T15:30:35.367 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:34] ENGINE Bus STARTING 2024-04-14T15:30:35.367 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:35] ENGINE Serving on http://:::9283 2024-04-14T15:30:35.367 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:35] ENGINE Bus STARTED 2024-04-14T15:30:35.367 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:35] ENGINE Bus STOPPING 2024-04-14T15:30:35.368 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:30:35.368 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:35] ENGINE Bus STOPPED 2024-04-14T15:30:35.368 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:35] ENGINE Bus STARTING 2024-04-14T15:30:35.368 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:35] ENGINE Serving on http://:::9283 2024-04-14T15:30:35.368 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:30:35] ENGINE Bus STARTED 2024-04-14T15:30:35.664 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:35 smithi196 podman[51180]: 2024-04-14 15:30:35.367735894 +0000 UTC m=+0.658977233 container remove 848a2eb010e78f921444581433b15eececa9e6122574e526c111e309327ae6a7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, io.openshift.expose-services=, release=236.1648460182, architecture=x86_64, io.buildah.version=1.24.2, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, maintainer=Paul Cuzner , description=Ceph Grafana Container) 2024-04-14T15:30:35.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:35 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:30:35.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:35 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:30:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:30:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:30:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:30:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:30:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[29517]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:30:36.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:30:36.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:30:36.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:30:36.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:30:36.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:36.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:36.114 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:35 smithi016 ceph-mon[24869]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:30:36.165 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:30:35] "GET /metrics HTTP/1.1" 200 38143 "" "Prometheus/2.33.4" 2024-04-14T15:30:36.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:30:36.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T15:30:36.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:30:36.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T15:30:36.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:36.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:35 smithi196 ceph-mon[25696]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:30:37.775 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[29517]: Upgrade: Updating mgr.y 2024-04-14T15:30:37.775 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:37.775 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:30:37.776 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:30:37.776 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:30:37.776 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:37.776 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[29517]: Deploying daemon mgr.y on smithi016 2024-04-14T15:30:37.776 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[24869]: Upgrade: Updating mgr.y 2024-04-14T15:30:37.776 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:37.776 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:30:37.776 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:30:37.776 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:30:37.776 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:37.776 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:37 smithi016 ceph-mon[24869]: Deploying daemon mgr.y on smithi016 2024-04-14T15:30:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:37 smithi196 ceph-mon[25696]: Upgrade: Updating mgr.y 2024-04-14T15:30:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:37 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:37 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:30:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:37 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:30:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:37 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:30:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:37 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:37 smithi196 ceph-mon[25696]: Deploying daemon mgr.y on smithi016 2024-04-14T15:30:38.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:38 smithi016 systemd[1]: Stopping Ceph mgr.y for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:38.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:38 smithi016 ceph-mon[29517]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-04-14T15:30:38.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:38 smithi016 ceph-mon[24869]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-04-14T15:30:38.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:38 smithi196 ceph-mon[25696]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-04-14T15:30:38.897 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:38 smithi016 podman[60310]: 2024-04-14 15:30:38.626603596 +0000 UTC m=+0.148086859 container died 8678f197809c666b732a6b6920caf13da144ca0f1ac833279896903d065f8fd5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y, vendor=Red Hat, Inc., io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , ceph=True, name=centos-stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, version=8, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, distribution-scope=public, io.openshift.tags=base centos centos-stream) 2024-04-14T15:30:38.914 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:30:39.230 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:38 smithi016 podman[60310]: 2024-04-14 15:30:38.99618406 +0000 UTC m=+0.517667322 container cleanup 8678f197809c666b732a6b6920caf13da144ca0f1ac833279896903d065f8fd5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y, CEPH_POINT_RELEASE=-17.2.0, version=8, ceph=True, io.buildah.version=1.19.8, GIT_CLEAN=True, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, distribution-scope=public, vendor=Red Hat, Inc., release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-14T15:30:39.231 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:38 smithi016 bash[60310]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y 2024-04-14T15:30:39.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:39 smithi016 podman[60324]: 2024-04-14 15:30:39.23111991 +0000 UTC m=+0.605151892 container remove 8678f197809c666b732a6b6920caf13da144ca0f1ac833279896903d065f8fd5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, com.redhat.component=centos-stream-container, distribution-scope=public, io.buildah.version=1.19.8, version=8, release=754, maintainer=Guillaume Abrioux , RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-14T15:30:39.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:39 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-14T15:30:39.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:39 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-04-14T15:30:39.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:39 smithi016 systemd[1]: Stopped Ceph mgr.y for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:39.489 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:39 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.y.service: Consumed 57.324s CPU time. 2024-04-14T15:30:39.772 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:39 smithi016 systemd[1]: Starting Ceph mgr.y for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:39 smithi016 ceph-mon[24869]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:39 smithi016 ceph-mon[24869]: from='client.25114 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:40.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:39 smithi016 podman[60454]: 2024-04-14 15:30:39.715320583 +0000 UTC m=+0.018111291 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:30:40.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:39 smithi016 podman[60454]: 2024-04-14 15:30:39.815642127 +0000 UTC m=+0.118432856 container create e9559f3661299c2da10556bb2182e44f64dab37ac9bea69c17b6808464996dd9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-04-14T15:30:40.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:39 smithi016 ceph-mon[29517]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:40.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:39 smithi016 ceph-mon[29517]: from='client.25114 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:40.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:39 smithi196 ceph-mon[25696]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:40.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:39 smithi196 ceph-mon[25696]: from='client.25114 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:40.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:40 smithi016 podman[60454]: 2024-04-14 15:30:40.184979291 +0000 UTC m=+0.487769982 container init e9559f3661299c2da10556bb2182e44f64dab37ac9bea69c17b6808464996dd9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, ceph=True) 2024-04-14T15:30:40.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:40 smithi016 podman[60454]: 2024-04-14 15:30:40.187898578 +0000 UTC m=+0.490689263 container start e9559f3661299c2da10556bb2182e44f64dab37ac9bea69c17b6808464996dd9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-14T15:30:40.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:40 smithi016 ceph-mgr[60472]: -- 172.21.15.16:0/259550603 <== mon.1 v2:172.21.15.16:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c0d2f405a0 con 0x55c0d2f15000 2024-04-14T15:30:40.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:40 smithi016 bash[60454]: e9559f3661299c2da10556bb2182e44f64dab37ac9bea69c17b6808464996dd9 2024-04-14T15:30:40.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:40 smithi016 systemd[1]: Started Ceph mgr.y for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:41.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:40 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:40.760+0000 7fce83e42200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:30:41.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:40 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:40.862+0000 7fce83e42200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:30:41.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:41.582+0000 7fce83e42200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:30:41.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:41 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:41 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:41 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:41 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:41 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:41 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:41.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:41.678+0000 7fce83e42200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:30:41.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:41.828+0000 7fce83e42200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:30:41.931 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:41.930+0000 7fce83e42200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:30:42.214 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:42.018+0000 7fce83e42200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:30:42.486 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:42 smithi016 ceph-mon[29517]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:42.487 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:42 smithi016 ceph-mon[24869]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:42.487 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:42.315+0000 7fce83e42200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:30:42.487 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:42.437+0000 7fce83e42200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:30:42.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:42 smithi196 ceph-mon[25696]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:42.743 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:42.533+0000 7fce83e42200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:30:42.743 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:42.643+0000 7fce83e42200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:30:43.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:42.824+0000 7fce83e42200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:30:43.914 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:30:44.339 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:43.897+0000 7fce83e42200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:30:44.339 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:44.033+0000 7fce83e42200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:30:44.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:44 smithi016 ceph-mon[29517]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:44.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:44 smithi016 ceph-mon[24869]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:44.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:44.393+0000 7fce83e42200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:30:44.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:44.491+0000 7fce83e42200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:30:44.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:44 smithi196 ceph-mon[25696]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:45.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:44.699+0000 7fce83e42200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:30:45.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:44.921+0000 7fce83e42200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:30:45.494 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:45 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:45.225+0000 7fce83e42200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:30:45.495 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:45 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:45.495 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:45 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:45.495 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:45 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:45.495 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:45 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:45.495 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:45 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:45.495 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:45 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:45 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:45 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:45.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:45 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:45.645 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:45 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-04-14T15:30:45.645 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:45 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:45.645 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:45 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:45.752 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:45 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:45.494+0000 7fce83e42200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:30:45.914 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:30:45] "GET /metrics HTTP/1.1" 200 38147 "" "Prometheus/2.33.4" 2024-04-14T15:30:45.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:45 smithi196 podman[51906]: 2024-04-14 15:30:45.546968959 +0000 UTC m=+0.017141873 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:30:45.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:45 smithi196 podman[51906]: 2024-04-14 15:30:45.691252687 +0000 UTC m=+0.161425645 container create 97c7e838ab9adeb227e617083673c99efbe352ed0d2a993efc565b48fb63c45b (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.buildah.version=1.24.2, release=236.1648460182, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.openshift.expose-services=, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-type=git, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-14T15:30:46.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:45 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:45.751+0000 7fce83e42200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:30:46.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:45 smithi196 podman[51906]: 2024-04-14 15:30:45.954031488 +0000 UTC m=+0.424204388 container init 97c7e838ab9adeb227e617083673c99efbe352ed0d2a993efc565b48fb63c45b (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, release=236.1648460182, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, description=Ceph Grafana Container, io.openshift.tags=base rhel8, io.openshift.expose-services=, name=ubi8, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-14T15:30:46.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:45 smithi196 podman[51906]: 2024-04-14 15:30:45.956779112 +0000 UTC m=+0.426952014 container start 97c7e838ab9adeb227e617083673c99efbe352ed0d2a993efc565b48fb63c45b (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.openshift.expose-services=, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, architecture=x86_64, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, vendor=Red Hat, Inc., vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-14T15:30:46.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[51921]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:30:46.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[51921]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:30:46.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:46 smithi196 bash[51906]: 97c7e838ab9adeb227e617083673c99efbe352ed0d2a993efc565b48fb63c45b 2024-04-14T15:30:46.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:46 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:46.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:46 smithi196 podman[51937]: 2024-04-14 15:30:46.087846338 +0000 UTC m=+0.096976610 container died 97c7e838ab9adeb227e617083673c99efbe352ed0d2a993efc565b48fb63c45b (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, vcs-type=git, distribution-scope=public, version=8.5, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., release=236.1648460182, description=Ceph Grafana Container, io.openshift.expose-services=, maintainer=Paul Cuzner , architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2) 2024-04-14T15:30:46.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:46 smithi196 ceph-mon[25696]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:46.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:46 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:46.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:46 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:46.917 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:46 smithi196 podman[51937]: 2024-04-14 15:30:46.744657021 +0000 UTC m=+0.753787318 container remove 97c7e838ab9adeb227e617083673c99efbe352ed0d2a993efc565b48fb63c45b (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, vcs-type=git, vendor=Red Hat, Inc., release=236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , architecture=x86_64, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5) 2024-04-14T15:30:46.917 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:46 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:30:46.917 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:46 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:30:47.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:46 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:46.631+0000 7fce83e42200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:30:47.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:46 smithi016 ceph-mon[24869]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:47.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:46 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:47.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:46 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:47.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:46 smithi016 ceph-mon[29517]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:30:47.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:46 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:47.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:46 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:47.538 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:47.238+0000 7fce83e42200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:30:47.539 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:47.351+0000 7fce83e42200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:30:47.789 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:47.789 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:47.789 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[24869]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:47.789 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[24869]: Standby manager daemon y restarted 2024-04-14T15:30:47.790 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[24869]: Standby manager daemon y started 2024-04-14T15:30:47.790 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T15:30:47.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:47.584+0000 7fce83e42200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:30:47.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:30:47] ENGINE Bus STARTING 2024-04-14T15:30:47.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: CherryPy Checker: 2024-04-14T15:30:47.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: The Application mounted at '' has an empty config. 2024-04-14T15:30:47.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:30:47.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:30:47] ENGINE Serving on http://:::9283 2024-04-14T15:30:47.790 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:30:47] ENGINE Bus STARTED 2024-04-14T15:30:48.096 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:48.096 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:48.097 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[29517]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:48.097 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[29517]: Standby manager daemon y restarted 2024-04-14T15:30:48.097 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[29517]: Standby manager daemon y started 2024-04-14T15:30:48.097 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T15:30:48.097 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:30:48.097 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T15:30:48.097 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:30:48.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:30:48.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T15:30:48.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:47 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:30:48.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:47 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:48.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:47 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:48.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:47 smithi196 ceph-mon[25696]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:48.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:47 smithi196 ceph-mon[25696]: Standby manager daemon y restarted 2024-04-14T15:30:48.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:47 smithi196 ceph-mon[25696]: Standby manager daemon y started 2024-04-14T15:30:48.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:47 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T15:30:48.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:47 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:30:48.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:47 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T15:30:48.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:47 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:30:48.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:48 smithi196 ceph-mon[25696]: mgrmap e28: x(active, since 79s), standbys: y 2024-04-14T15:30:48.916 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:30:49.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:48 smithi016 ceph-mon[24869]: mgrmap e28: x(active, since 79s), standbys: y 2024-04-14T15:30:49.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:48 smithi016 ceph-mon[29517]: mgrmap e28: x(active, since 79s), standbys: y 2024-04-14T15:30:50.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:49 smithi016 ceph-mon[24869]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:49 smithi016 ceph-mon[24869]: from='client.25114 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:49 smithi016 ceph-mon[29517]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:49 smithi016 ceph-mon[29517]: from='client.25114 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:50.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:49 smithi196 ceph-mon[25696]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:30:50.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:49 smithi196 ceph-mon[25696]: from='client.25114 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:30:51.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:51.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:51.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:51.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:30:51.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[29517]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:30:51.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:51.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[29517]: Reconfiguring iscsi.foo.smithi016.sijcrc (dependencies changed)... 2024-04-14T15:30:51.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:51.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:51.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:51.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[29517]: Reconfiguring daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:30:51.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:51.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:51.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:51.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:30:51.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[24869]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:30:51.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:51.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[24869]: Reconfiguring iscsi.foo.smithi016.sijcrc (dependencies changed)... 2024-04-14T15:30:51.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:51.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:51.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:51.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:51 smithi016 ceph-mon[24869]: Reconfiguring daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:30:51.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:51 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:51.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:51 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:51.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:51 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:51.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:51 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:30:51.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:51 smithi196 ceph-mon[25696]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:30:51.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:51 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' 2024-04-14T15:30:51.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:51 smithi196 ceph-mon[25696]: Reconfiguring iscsi.foo.smithi016.sijcrc (dependencies changed)... 2024-04-14T15:30:51.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:51 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:51.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:51 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:30:51.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:51 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:30:51.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:51 smithi196 ceph-mon[25696]: Reconfiguring daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:30:52.651 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:52 smithi016 ceph-mon[29517]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:30:52.651 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:52 smithi016 ceph-mon[24869]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:30:52.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:52 smithi196 ceph-mon[25696]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:30:53.488 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:53 smithi016 ceph-mon[29517]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-04-14T15:30:53.489 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:53 smithi016 ceph-mon[24869]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-04-14T15:30:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:53 smithi196 ceph-mon[25696]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-04-14T15:30:53.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:30:53 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:30:53.506Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:30:54.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:53.999+0000 7fa023fcd700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:54.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Creating ceph-iscsi config... 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Traceback (most recent call last): 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: return _run_code(code, main_globals, None, 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: exec(code, run_globals) 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:54.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:54.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:54.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:54.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:54.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:54.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:54.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Traceback (most recent call last): 2024-04-14T15:30:54.417 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: return self.event_loop.get_result(coro, timeout) 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: return future.result(timeout) 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: return self.__get_result() 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: raise self._exception 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: config_blobs=daemon_spec.final_config, 2024-04-14T15:30:54.418 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Creating ceph-iscsi config... 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: Traceback (most recent call last): 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: return _run_code(code, main_globals, None, 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: exec(code, run_globals) 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:54.419 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:54.420 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:54.420 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:54.420 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:54.420 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:54.420 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:54.420 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:54.420 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[24869]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: exec(code, run_globals) 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T15:30:55.112 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: return future.result(timeout) 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: return self.__get_result() 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: raise self._exception 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T15:30:55.113 INFO:journalctl@ceph.mon.a.smithi016.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: exec(code, run_globals) 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:55.114 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:55.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[29517]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:55.115 INFO:journalctl@ceph.mon.c.smithi016.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:55.115 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:55.115 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:55.115 INFO:journalctl@ceph.mon.c.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:55.115 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.115 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.115 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:55.115 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:55.115 INFO:journalctl@ceph.mon.c.smithi016.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: exec(code, run_globals) 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T15:30:55.116 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: return future.result(timeout) 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: return self.__get_result() 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: raise self._exception 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:55.117 INFO:journalctl@ceph.mon.c.smithi016.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: Traceback (most recent call last): 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: exec(code, run_globals) 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:55.118 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:55.119 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:30:54] ENGINE Bus STOPPING 2024-04-14T15:30:55.119 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:30:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:30:55.119 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:30:54] ENGINE Bus STOPPED 2024-04-14T15:30:55.119 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:30:54] ENGINE Bus STARTING 2024-04-14T15:30:55.119 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:30:54] ENGINE Serving on http://:::9283 2024-04-14T15:30:55.119 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:30:54] ENGINE Bus STARTED 2024-04-14T15:30:55.119 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:55.119 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.a.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.a.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.a.smithi016.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[24869]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[24869]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[24869]: Failing over to other MGR 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[24869]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[24869]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[24869]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:55.120 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:55.121 INFO:journalctl@ceph.mon.c.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:55.121 INFO:journalctl@ceph.mon.c.smithi016.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.121 INFO:journalctl@ceph.mon.c.smithi016.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.121 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:30:55.121 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[29517]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:55.121 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[29517]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:55.121 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[29517]: Failing over to other MGR 2024-04-14T15:30:55.121 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[29517]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T15:30:55.121 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T15:30:55.121 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[29517]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T15:30:55.121 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:54 smithi016 ceph-mon[29517]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-04-14T15:30:55.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-mon[25696]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:55.165 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:55.165 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:55.165 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:55.165 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:55.165 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.166 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.166 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-04-14T15:30:55.166 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:55.166 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:55.166 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:55.166 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-04-14T15:30:55.166 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:55.166 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:55.166 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:55.166 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: return future.result(timeout) 2024-04-14T15:30:55.167 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: return self.__get_result() 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: raise self._exception 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: Creating ceph-iscsi config... 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: Non-zero exit code 1 from systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.168 INFO:journalctl@ceph.mon.b.smithi196.stdout: systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.169 INFO:journalctl@ceph.mon.b.smithi196.stdout: Traceback (most recent call last): 2024-04-14T15:30:55.169 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:30:55.169 INFO:journalctl@ceph.mon.b.smithi196.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:30:55.169 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:30:55.169 INFO:journalctl@ceph.mon.b.smithi196.stdout: exec(code, run_globals) 2024-04-14T15:30:55.169 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:30:55.169 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:30:55.169 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:30:55.169 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:30:55.169 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:54.781+0000 7fa09195a700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: ignoring --setuser ceph since I am not root 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: ignoring --setgroup ceph since I am not root 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-mgr[46927]: -- 172.21.15.196:0/2404975582 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b780eec5a0 con 0x55b780ec0800 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mon.b.smithi196.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mon.b.smithi196.stdout: RuntimeError: Failed command: systemctl restart ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mon.b.smithi196.stdout: See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-mon[25696]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-mon[25696]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T15:30:55.170 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-mon[25696]: Failing over to other MGR 2024-04-14T15:30:55.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-mon[25696]: from='mgr.34865 172.21.15.196:0/1685021253' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T15:30:55.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T15:30:55.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-mon[25696]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T15:30:55.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:54 smithi196 ceph-mon[25696]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-04-14T15:30:55.509 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:55.410+0000 7f2dff2da200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:30:55.509 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:55.508+0000 7f2dff2da200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.34865 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: mgrmap e29: y(active, starting, since 0.768529s) 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:30:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:30:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:30:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:30:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:30:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:30:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: Manager daemon y is now available 2024-04-14T15:30:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:30:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:30:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:30:56.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.34865 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-14T15:30:56.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: mgrmap e29: y(active, starting, since 0.768529s) 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:30:56.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:30:56.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: Manager daemon y is now available 2024-04-14T15:30:56.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:30:56.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:56.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:30:56.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:55 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.34865 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: mgrmap e29: y(active, starting, since 0.768529s) 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:30:56.147 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:30:56.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:30:56.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:30:56.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:30:56.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:30:56.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:30:56.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: Manager daemon y is now available 2024-04-14T15:30:56.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:30:56.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:30:56.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:30:56.148 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:55 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:30:56.414 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:56.215+0000 7f2dff2da200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:30:56.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:56.310+0000 7f2dff2da200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:30:56.766 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:56.455+0000 7f2dff2da200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:30:56.766 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:56.565+0000 7f2dff2da200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:30:56.766 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:56.648+0000 7f2dff2da200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:30:56.963 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:56 smithi016 ceph-mon[29517]: mgrmap e30: y(active, since 1.77535s) 2024-04-14T15:30:56.963 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:56 smithi016 ceph-mon[29517]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:30:56.964 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:56 smithi016 ceph-mon[24869]: mgrmap e30: y(active, since 1.77535s) 2024-04-14T15:30:56.964 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:56 smithi016 ceph-mon[24869]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:30:57.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:56 smithi196 ceph-mon[25696]: mgrmap e30: y(active, since 1.77535s) 2024-04-14T15:30:57.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:56 smithi196 ceph-mon[25696]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:30:57.019 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:56.927+0000 7f2dff2da200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:30:57.019 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:56 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-04-14T15:30:57.019 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:56 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:57.019 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:56 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:30:57.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:57.047+0000 7f2dff2da200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:30:57.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:57.134+0000 7f2dff2da200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:30:57.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:57.230+0000 7f2dff2da200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:30:57.790 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:57.438+0000 7f2dff2da200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:30:57.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:57 smithi196 podman[52369]: 2024-04-14 15:30:57.392572817 +0000 UTC m=+0.019929034 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:30:57.791 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:57 smithi196 podman[52369]: 2024-04-14 15:30:57.520242645 +0000 UTC m=+0.147598843 container create d7fa9f168321b9069f0518c215f5732f7c93a5283f71540605a56288e8b43b15 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, maintainer=Paul Cuzner , io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, architecture=x86_64, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-14T15:30:58.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[24869]: [14/Apr/2024:15:30:56] ENGINE Bus STARTING 2024-04-14T15:30:58.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[24869]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[24869]: Cluster is now healthy 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[24869]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[24869]: [14/Apr/2024:15:30:56] ENGINE Serving on http://172.21.15.16:8765 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[24869]: [14/Apr/2024:15:30:56] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[24869]: [14/Apr/2024:15:30:56] ENGINE Bus STARTED 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[29517]: [14/Apr/2024:15:30:56] ENGINE Bus STARTING 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[29517]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[29517]: Cluster is now healthy 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[29517]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[29517]: [14/Apr/2024:15:30:56] ENGINE Serving on http://172.21.15.16:8765 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[29517]: [14/Apr/2024:15:30:56] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:30:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:57 smithi016 ceph-mon[29517]: [14/Apr/2024:15:30:56] ENGINE Bus STARTED 2024-04-14T15:30:58.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:57 smithi196 ceph-mon[25696]: [14/Apr/2024:15:30:56] ENGINE Bus STARTING 2024-04-14T15:30:58.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:57 smithi196 ceph-mon[25696]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T15:30:58.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:57 smithi196 ceph-mon[25696]: Cluster is now healthy 2024-04-14T15:30:58.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:57 smithi196 ceph-mon[25696]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:30:58.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:57 smithi196 ceph-mon[25696]: [14/Apr/2024:15:30:56] ENGINE Serving on http://172.21.15.16:8765 2024-04-14T15:30:58.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:57 smithi196 ceph-mon[25696]: [14/Apr/2024:15:30:56] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:30:58.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:57 smithi196 ceph-mon[25696]: [14/Apr/2024:15:30:56] ENGINE Bus STARTED 2024-04-14T15:30:58.796 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:58.434+0000 7f2dff2da200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:30:58.796 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:58.566+0000 7f2dff2da200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:30:58.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:58 smithi196 podman[52369]: 2024-04-14 15:30:58.577624375 +0000 UTC m=+1.204980575 container init d7fa9f168321b9069f0518c215f5732f7c93a5283f71540605a56288e8b43b15 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, name=ubi8, io.openshift.expose-services=, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-14T15:30:58.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:58 smithi196 podman[52369]: 2024-04-14 15:30:58.581183021 +0000 UTC m=+1.208539229 container start d7fa9f168321b9069f0518c215f5732f7c93a5283f71540605a56288e8b43b15 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, architecture=x86_64, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.openshift.expose-services=, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, vcs-type=git, io.buildah.version=1.24.2) 2024-04-14T15:30:58.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[52408]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:30:58.797 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[52408]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:30:59.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:58 smithi016 ceph-mon[29517]: mgrmap e31: y(active, since 3s) 2024-04-14T15:30:59.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:58 smithi016 ceph-mon[24869]: mgrmap e31: y(active, since 3s) 2024-04-14T15:30:59.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:58 smithi196 ceph-mon[25696]: mgrmap e31: y(active, since 3s) 2024-04-14T15:30:59.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:58.915+0000 7f2dff2da200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:30:59.139 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:59.010+0000 7f2dff2da200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:30:59.409 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:59.218+0000 7f2dff2da200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:30:59.413 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:30:59.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:59.421+0000 7f2dff2da200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:30:59.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:59 smithi196 bash[52369]: d7fa9f168321b9069f0518c215f5732f7c93a5283f71540605a56288e8b43b15 2024-04-14T15:30:59.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:59 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:30:59.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:30:59 smithi196 podman[52454]: 2024-04-14 15:30:59.469595319 +0000 UTC m=+0.854164059 container died d7fa9f168321b9069f0518c215f5732f7c93a5283f71540605a56288e8b43b15 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, distribution-scope=public, release=236.1648460182, architecture=x86_64, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., maintainer=Paul Cuzner ) 2024-04-14T15:30:59.992 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:59.726+0000 7f2dff2da200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (56s) 1s ago 5m 15.0M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 unknown 15s ago 5m - - 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 unknown 1s ago 4m - - 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283 running (100s) 15s ago 10m 515M - 19.0.0-2778-ga971fd5f b96c509eef84 ef0a2cd931a5 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (20s) 1s ago 11m 474M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (11m) 1s ago 12m 54.2M 2048M 17.2.0 e1d6a67b021e e8cfd2da2fd9 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (10m) 15s ago 10m 41.6M 2048M 17.2.0 e1d6a67b021e f933c5fa37fd 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (10m) 1s ago 10m 43.9M 2048M 17.2.0 e1d6a67b021e 8629057bb8e9 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (66s) 1s ago 6m 9.99M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (38s) 15s ago 6m 10.4M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (10m) 1s ago 10m 52.4M 2737M 17.2.0 e1d6a67b021e b812f3e0fa81 2024-04-14T15:31:00.124 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (9m) 1s ago 9m 51.9M 2737M 17.2.0 e1d6a67b021e c275d1a0eaeb 2024-04-14T15:31:00.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (9m) 1s ago 9m 50.1M 2737M 17.2.0 e1d6a67b021e 592ce301a74a 2024-04-14T15:31:00.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (8m) 1s ago 8m 50.6M 2737M 17.2.0 e1d6a67b021e 1ed8730c3fc1 2024-04-14T15:31:00.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (8m) 15s ago 8m 53.1M 2992M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:31:00.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (8m) 15s ago 8m 49.9M 2992M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:31:00.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (7m) 15s ago 7m 49.2M 2992M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:31:00.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (7m) 15s ago 7m 53.3M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:31:00.125 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (40s) 15s ago 6m 54.7M - 2.33.4 514e6a882f6e c4255d134085 2024-04-14T15:31:00.125 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (5m) 1s ago 5m 73.3M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:31:00.125 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (5m) 15s ago 5m 73.2M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:31:00.125 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (4m) 1s ago 4m 71.4M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:31:00.125 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (4m) 15s ago 4m 73.5M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:31:00.252 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:30:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:30:59.997+0000 7f2dff2da200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:31:00.253 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:59 smithi196 ceph-mon[25696]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:00.253 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:59 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:00.253 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:30:59 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:00.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:59 smithi016 ceph-mon[29517]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:00.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:59 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:00.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:30:59 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:00.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:59 smithi016 ceph-mon[24869]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:00.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:59 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:00.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:30:59 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:00.543 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:31:00.543 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:31:00.543 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T15:31:00.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:31:00.543 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:31:00.543 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T15:31:00.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:31:00.543 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:31:00.543 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T15:31:00.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:31:00.543 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": {}, 2024-04-14T15:31:00.544 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:31:00.544 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:31:00.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:31:00.544 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:31:00.544 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-04-14T15:31:00.544 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T15:31:00.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:31:00.544 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:31:00.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:00 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:31:00.251+0000 7f2dff2da200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:31:00.913 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:31:00.913 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:31:00.913 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:31:00.913 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:31:00.913 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:31:00.913 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-14T15:31:00.914 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:31:00.914 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "2/23 daemons upgraded", 2024-04-14T15:31:00.914 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "", 2024-04-14T15:31:00.914 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:31:00.914 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:31:01.333 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_OK 2024-04-14T15:31:01.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[24869]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:01.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[24869]: from='client.25246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:01.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[24869]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:01.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:01.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:01.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/824337742' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:01.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[29517]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:01.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[29517]: from='client.25246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:01.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[29517]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:01.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:01.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:01.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:01 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/824337742' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:01.414 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:31:01.079+0000 7f2dff2da200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:31:01.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:01 smithi196 ceph-mon[25696]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:01.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:01 smithi196 ceph-mon[25696]: from='client.25246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:01.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:01 smithi196 ceph-mon[25696]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:01.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:01 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:01.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:01 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:01.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:01 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/824337742' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:01.914 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:31:01.663+0000 7f2dff2da200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:31:01.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:31:01.765+0000 7f2dff2da200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:31:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[24869]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[24869]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/1272504926' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:31:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[24869]: Standby manager daemon x started 2024-04-14T15:31:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:31:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:31:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:31:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:31:02.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[29517]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:02.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[29517]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:02.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/1272504926' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:31:02.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[29517]: Standby manager daemon x started 2024-04-14T15:31:02.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:31:02.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:31:02.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:31:02.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:02 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:31:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-mon[25696]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-mon[25696]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/1272504926' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:31:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-mon[25696]: Standby manager daemon x started 2024-04-14T15:31:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:31:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:31:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:31:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/1844425656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:31:02.416 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:02 smithi196 podman[52454]: 2024-04-14 15:31:02.132209843 +0000 UTC m=+3.516778569 container remove d7fa9f168321b9069f0518c215f5732f7c93a5283f71540605a56288e8b43b15 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.buildah.version=1.24.2, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-14T15:31:02.416 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:02 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:31:02.416 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:02 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:31:02.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:31:01.987+0000 7f2dff2da200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:31:02.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:31:01] ENGINE Bus STARTING 2024-04-14T15:31:02.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: CherryPy Checker: 2024-04-14T15:31:02.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: The Application mounted at '' has an empty config. 2024-04-14T15:31:02.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: 2024-04-14T15:31:02.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:31:02] ENGINE Serving on http://:::9283 2024-04-14T15:31:02.416 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: [14/Apr/2024:15:31:02] ENGINE Bus STARTED 2024-04-14T15:31:03.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:03 smithi196 ceph-mon[25696]: mgrmap e32: y(active, since 8s), standbys: x 2024-04-14T15:31:03.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:03 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:31:03.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:03 smithi016 ceph-mon[29517]: mgrmap e32: y(active, since 8s), standbys: x 2024-04-14T15:31:03.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:03 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:31:03.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:03 smithi016 ceph-mon[24869]: mgrmap e32: y(active, since 8s), standbys: x 2024-04-14T15:31:03.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:03 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:31:03.914 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:03.507Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:31:04.303 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:04 smithi016 ceph-mon[29517]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-14T15:31:04.303 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:04 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:04.303 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:04 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:04.303 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:04 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:04.303 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:04 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:04.303 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:04 smithi016 ceph-mon[24869]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-14T15:31:04.303 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:04 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:04.304 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:04 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:04.304 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:04 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:04.304 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:04 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:04.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:04 smithi196 ceph-mon[25696]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-14T15:31:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:04 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:04 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:04 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:04 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:06.164 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[46923]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:31:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T15:31:06.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:06 smithi196 ceph-mon[25696]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T15:31:06.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:06 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:06.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:06 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:06.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:06 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:06.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:06 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:31:06.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:06 smithi196 ceph-mon[25696]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:31:06.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:06 smithi196 ceph-mon[25696]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:31:06.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:06 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/3421604327' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:31:06.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[29517]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T15:31:06.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:06.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:06.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:06.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:31:06.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[29517]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:31:06.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[29517]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:31:06.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/3421604327' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:31:06.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[24869]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T15:31:06.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:06.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:06.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:06.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:31:06.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[24869]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:31:06.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[24869]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:31:06.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:06 smithi016 ceph-mon[24869]: from='client.? 172.21.15.16:0/3421604327' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:31:07.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.307 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:07 smithi196 ceph-mon[25696]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:31:07.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:07.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:07.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:07.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:07.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:07.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:07.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[29517]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T15:31:07.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:07.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:07 smithi016 ceph-mon[24869]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T15:31:08.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:08 smithi196 ceph-mon[25696]: Reconfiguring daemon prometheus.a on smithi196 2024-04-14T15:31:08.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:08 smithi196 ceph-mon[25696]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:31:08.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:08 smithi016 ceph-mon[29517]: Reconfiguring daemon prometheus.a on smithi196 2024-04-14T15:31:08.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:08 smithi016 ceph-mon[29517]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:31:08.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:08 smithi016 ceph-mon[24869]: Reconfiguring daemon prometheus.a on smithi196 2024-04-14T15:31:08.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:08 smithi016 ceph-mon[24869]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:31:09.409 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-mon[25696]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:31:09.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:09 smithi016 ceph-mon[29517]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:31:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:09 smithi016 ceph-mon[24869]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:31:09.988 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 systemd[1]: Stopping Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:09.988 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 bash[53943]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T15:31:10.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:10 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:31:10.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.988Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T15:31:10.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.988Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-14T15:31:10.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.988Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-14T15:31:10.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.988Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-14T15:31:10.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.988Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-14T15:31:10.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.988Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-14T15:31:10.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.988Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T15:31:10.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.988Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-14T15:31:10.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.988Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T15:31:10.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.989Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T15:31:10.315 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.989Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-14T15:31:10.315 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[50513]: ts=2024-04-14T15:31:09.989Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-14T15:31:10.315 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:09 smithi196 podman[53956]: 2024-04-14 15:31:09.998638399 +0000 UTC m=+0.115333687 container died c4255d1340850c0eebd5a88f911ac040da4e6c088f8be10dfd2cd19416b5a052 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:31:10.605 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:10 smithi196 podman[53956]: 2024-04-14 15:31:10.421794787 +0000 UTC m=+0.538490075 container cleanup c4255d1340850c0eebd5a88f911ac040da4e6c088f8be10dfd2cd19416b5a052 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:31:10.606 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:10 smithi196 bash[53956]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T15:31:10.606 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:10 smithi196 bash[53987]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T15:31:10.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:10 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:31:10.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:10 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:31:10.865 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:10 smithi196 podman[53971]: 2024-04-14 15:31:10.606306069 +0000 UTC m=+0.609669330 container remove c4255d1340850c0eebd5a88f911ac040da4e6c088f8be10dfd2cd19416b5a052 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:31:10.865 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:10 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-14T15:31:10.865 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:10 smithi196 systemd[1]: Stopped Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:10.865 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:10 smithi196 systemd[1]: Starting Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:10.865 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:10 smithi196 podman[54053]: 2024-04-14 15:31:10.741581895 +0000 UTC m=+0.018765798 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-14T15:31:10.865 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:10 smithi196 podman[54053]: 2024-04-14 15:31:10.865691177 +0000 UTC m=+0.142875109 container create 9f60db35d8a82abdbf6712da25a17e32c1047b2e8da6e2f7ea94d11b65707945 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:31:11.133 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 podman[54053]: 2024-04-14 15:31:11.084560811 +0000 UTC m=+0.361744702 container init 9f60db35d8a82abdbf6712da25a17e32c1047b2e8da6e2f7ea94d11b65707945 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:31:11.133 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 podman[54053]: 2024-04-14 15:31:11.0870963 +0000 UTC m=+0.364280195 container start 9f60db35d8a82abdbf6712da25a17e32c1047b2e8da6e2f7ea94d11b65707945 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:31:11.133 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.110Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-14T15:31:11.133 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.110Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-14T15:31:11.133 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.110Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-14T15:31:11.133 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.110Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi196 (none))" 2024-04-14T15:31:11.133 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.110Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T15:31:11.133 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.110Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T15:31:11.133 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.112Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T15:31:11.134 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.112Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-14T15:31:11.134 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.113Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-14T15:31:11.134 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.115Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T15:31:11.134 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.115Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.689µs 2024-04-14T15:31:11.134 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.115Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T15:31:11.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-mon[25696]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:31:11.415 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 bash[54053]: 9f60db35d8a82abdbf6712da25a17e32c1047b2e8da6e2f7ea94d11b65707945 2024-04-14T15:31:11.415 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 systemd[1]: Started Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:11.415 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.321Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-04-14T15:31:11.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:11 smithi016 ceph-mon[29517]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:31:11.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:11 smithi016 ceph-mon[24869]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:31:11.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.557Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-04-14T15:31:11.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.772Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-04-14T15:31:11.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.772Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-04-14T15:31:11.797 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.772Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.54µs wal_replay_duration=656.868968ms total_replay_duration=656.914262ms 2024-04-14T15:31:11.798 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.776Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T15:31:11.798 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.776Z caller=main.go:947 level=info msg="TSDB started" 2024-04-14T15:31:11.798 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.776Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T15:31:11.798 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.796Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.67819ms db_storage=809ns remote_storage=1.285µs web_handler=311ns query_engine=648ns scrape=402.522µs scrape_sd=95.382µs notify=14.97µs notify_sd=16.554µs rules=18.847084ms 2024-04-14T15:31:11.798 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:11.796Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-14T15:31:12.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:12 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-04-14T15:31:12.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:12 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:12.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:12 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:12.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:12 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:12.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:12 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:12.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:12 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:31:12.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:12 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:31:12.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:12 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:12.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:12 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:12.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:12 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:31:12.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:12 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:31:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:12 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:12 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:12 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:31:12.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:12 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:31:13.031 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:12 smithi196 podman[54264]: 2024-04-14 15:31:12.910361201 +0000 UTC m=+0.115877592 container create fcd3eb63d371365b8d446dc889482aca60a629f436365216b763bae0662c291d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, release=236.1648460182, architecture=x86_64, version=8.5, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, name=ubi8, maintainer=Paul Cuzner , io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, vcs-type=git, vendor=Red Hat, Inc., io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container) 2024-04-14T15:31:13.032 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:12 smithi196 podman[54264]: 2024-04-14 15:31:12.813179369 +0000 UTC m=+0.018695775 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:31:13.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:13 smithi196 ceph-mon[25696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:31:13.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:13 smithi196 ceph-mon[25696]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:31:13.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:13 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:31:13.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:13 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:31:13.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:13 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:13.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:13 smithi196 podman[54264]: 2024-04-14 15:31:13.146706144 +0000 UTC m=+0.352222551 container init fcd3eb63d371365b8d446dc889482aca60a629f436365216b763bae0662c291d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.openshift.expose-services=) 2024-04-14T15:31:13.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:13 smithi196 podman[54264]: 2024-04-14 15:31:13.150630751 +0000 UTC m=+0.356147143 container start fcd3eb63d371365b8d446dc889482aca60a629f436365216b763bae0662c291d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, version=8.5, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, vcs-type=git, io.openshift.expose-services=, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, name=ubi8, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner ) 2024-04-14T15:31:13.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[54344]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:31:13.416 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[54344]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:31:13.416 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:13 smithi196 bash[54264]: fcd3eb63d371365b8d446dc889482aca60a629f436365216b763bae0662c291d 2024-04-14T15:31:13.416 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:13 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:13.416 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:13 smithi196 podman[54360]: 2024-04-14 15:31:13.237432971 +0000 UTC m=+0.052452729 container died fcd3eb63d371365b8d446dc889482aca60a629f436365216b763bae0662c291d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, maintainer=Paul Cuzner , version=8.5, vcs-type=git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, io.buildah.version=1.24.2, com.redhat.component=ubi8-container) 2024-04-14T15:31:13.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:13 smithi016 ceph-mon[29517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:31:13.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:13 smithi016 ceph-mon[29517]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:31:13.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:13 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:31:13.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:13 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:31:13.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:13 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:13.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:13 smithi016 ceph-mon[24869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T15:31:13.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:13 smithi016 ceph-mon[24869]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T15:31:13.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:13 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:31:13.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:13 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:31:13.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:13 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:13.841 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:31:14.157 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:13 smithi196 podman[54360]: 2024-04-14 15:31:13.842016599 +0000 UTC m=+0.657036359 container remove fcd3eb63d371365b8d446dc889482aca60a629f436365216b763bae0662c291d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, com.redhat.component=ubi8-container, io.openshift.expose-services=, io.buildah.version=1.24.2, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public) 2024-04-14T15:31:14.157 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:13 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:31:14.157 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:13 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:31:14.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:14 smithi196 ceph-mon[25696]: Upgrade: Updating mgr.x 2024-04-14T15:31:14.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:14 smithi196 ceph-mon[25696]: Deploying daemon mgr.x on smithi196 2024-04-14T15:31:14.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:14 smithi016 ceph-mon[29517]: Upgrade: Updating mgr.x 2024-04-14T15:31:14.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:14 smithi016 ceph-mon[29517]: Deploying daemon mgr.x on smithi196 2024-04-14T15:31:14.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:14 smithi016 ceph-mon[24869]: Upgrade: Updating mgr.x 2024-04-14T15:31:14.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:14 smithi016 ceph-mon[24869]: Deploying daemon mgr.x on smithi196 2024-04-14T15:31:15.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:15 smithi196 ceph-mon[25696]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 541 B/s rd, 0 op/s 2024-04-14T15:31:15.576 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:15 smithi196 systemd[1]: Stopping Ceph mgr.x for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:15.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:15 smithi016 ceph-mon[24869]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 541 B/s rd, 0 op/s 2024-04-14T15:31:15.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:15 smithi016 ceph-mon[29517]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 541 B/s rd, 0 op/s 2024-04-14T15:31:15.837 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:15 smithi196 podman[54492]: 2024-04-14 15:31:15.575513579 +0000 UTC m=+0.111370430 container died ef0a2cd931a529dbe55aba783ab9f49a145eab49e09c4d7e4612ec57eb6264c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-14T15:31:16.113 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:15 smithi196 podman[54492]: 2024-04-14 15:31:15.911413015 +0000 UTC m=+0.447269864 container cleanup ef0a2cd931a529dbe55aba783ab9f49a145eab49e09c4d7e4612ec57eb6264c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-14T15:31:16.113 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:15 smithi196 bash[54492]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x 2024-04-14T15:31:16.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:16 smithi196 ceph-mon[25696]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:16.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:16 smithi196 podman[54506]: 2024-04-14 15:31:16.113515043 +0000 UTC m=+0.537200700 container remove ef0a2cd931a529dbe55aba783ab9f49a145eab49e09c4d7e4612ec57eb6264c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=main-a971fd5) 2024-04-14T15:31:16.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:16 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-14T15:31:16.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:16 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-14T15:31:16.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:16 smithi196 systemd[1]: Stopped Ceph mgr.x for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:16.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:16 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.x.service: Consumed 28.004s CPU time. 2024-04-14T15:31:16.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:16 smithi016 ceph-mon[29517]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:16.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:16 smithi016 ceph-mon[24869]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:16.773 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:16 smithi196 systemd[1]: Starting Ceph mgr.x for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:16.773 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:16 smithi196 podman[54637]: 2024-04-14 15:31:16.633707147 +0000 UTC m=+0.017560913 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:31:17.035 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:16 smithi196 podman[54637]: 2024-04-14 15:31:16.772940154 +0000 UTC m=+0.156793958 container create c87a0109a594c505098fa34e957c65dcbc29e97395aa35289e516479b9ee6eff (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-14T15:31:17.035 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:17 smithi196 podman[54637]: 2024-04-14 15:31:17.008859582 +0000 UTC m=+0.392713351 container init c87a0109a594c505098fa34e957c65dcbc29e97395aa35289e516479b9ee6eff (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:31:17.035 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:17 smithi196 podman[54637]: 2024-04-14 15:31:17.011464939 +0000 UTC m=+0.395318707 container start c87a0109a594c505098fa34e957c65dcbc29e97395aa35289e516479b9ee6eff (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-04-14T15:31:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:17 smithi196 ceph-mon[25696]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 1 op/s 2024-04-14T15:31:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:17 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:17 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:17.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:17 smithi196 ceph-mgr[54656]: -- 172.21.15.196:0/3187627034 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558cd229c5a0 con 0x558cd2270c00 2024-04-14T15:31:17.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:17 smithi196 bash[54637]: c87a0109a594c505098fa34e957c65dcbc29e97395aa35289e516479b9ee6eff 2024-04-14T15:31:17.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:17 smithi196 systemd[1]: Started Ceph mgr.x for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:17.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:17 smithi016 ceph-mon[29517]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 1 op/s 2024-04-14T15:31:17.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:17 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:17.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:17 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:17.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:17 smithi016 ceph-mon[24869]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 1 op/s 2024-04-14T15:31:17.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:17 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:17.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:17 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:17.914 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:17 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:17.579+0000 7fc6899dd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:31:17.914 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:17 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:17.686+0000 7fc6899dd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:31:18.369 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:18 smithi196 ceph-mon[25696]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:31:18.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:18 smithi016 ceph-mon[24869]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:31:18.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:18 smithi016 ceph-mon[29517]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:31:18.622 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:18.368+0000 7fc6899dd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:31:18.623 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:18.462+0000 7fc6899dd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:31:18.903 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:18.622+0000 7fc6899dd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:31:18.903 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:18.720+0000 7fc6899dd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:31:18.903 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:18.804+0000 7fc6899dd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:31:19.164 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:19.083+0000 7fc6899dd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:31:19.596 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:19.196+0000 7fc6899dd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:31:19.596 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:19.297+0000 7fc6899dd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:31:19.597 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:19.409+0000 7fc6899dd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:31:19.597 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:19 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:19.596+0000 7fc6899dd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:31:20.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:20.603+0000 7fc6899dd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:31:21.106 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:20 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:20.737+0000 7fc6899dd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:31:21.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:21 smithi196 ceph-mon[25696]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:31:21.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:21 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:21.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:21 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:21.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:21.106+0000 7fc6899dd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:31:21.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:21.204+0000 7fc6899dd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:31:21.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:21 smithi016 ceph-mon[24869]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:31:21.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:21 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:21.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:21 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:21.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:21 smithi016 ceph-mon[29517]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:31:21.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:21 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:21.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:21 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:21.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:21 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:31:21] "GET /metrics HTTP/1.1" 200 38235 "" "Prometheus/2.33.4" 2024-04-14T15:31:21.914 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:21.417+0000 7fc6899dd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:31:21.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:21.618+0000 7fc6899dd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:31:22.211 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:21 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:21.936+0000 7fc6899dd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:31:22.471 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:22 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:22.211+0000 7fc6899dd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:31:22.914 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:22 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:22.471+0000 7fc6899dd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:31:23.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:23 smithi196 ceph-mon[25696]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:23.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:23 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:23.286 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:23 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:23.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:23 smithi016 ceph-mon[29517]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:23.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:23 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:23.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:23 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:23.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:23 smithi016 ceph-mon[24869]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:23.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:23 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:23.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:23 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:23.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:23.336+0000 7fc6899dd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:31:23.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:31:24.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:23 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-04-14T15:31:24.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:23 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:24.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:23 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:24.165 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:23.929+0000 7fc6899dd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:31:24.165 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:24.050+0000 7fc6899dd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:31:24.603 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:24.278+0000 7fc6899dd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:31:24.603 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: [14/Apr/2024:15:31:24] ENGINE Bus STARTING 2024-04-14T15:31:24.604 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: CherryPy Checker: 2024-04-14T15:31:24.604 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: The Application mounted at '' has an empty config. 2024-04-14T15:31:24.604 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:24.604 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: [14/Apr/2024:15:31:24] ENGINE Serving on http://:::9283 2024-04-14T15:31:24.604 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: [14/Apr/2024:15:31:24] ENGINE Bus STARTED 2024-04-14T15:31:24.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-mon[25696]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:31:24.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-mon[25696]: Standby manager daemon x restarted 2024-04-14T15:31:24.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-mon[25696]: Standby manager daemon x started 2024-04-14T15:31:24.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:31:24.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:31:24.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:31:24.881 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:31:24.882 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:24 smithi196 podman[55701]: 2024-04-14 15:31:24.50586672 +0000 UTC m=+0.017195221 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:31:24.882 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:24 smithi196 podman[55701]: 2024-04-14 15:31:24.626501108 +0000 UTC m=+0.137829609 container create aed2a23652fbe3f1e7674d4675a797aaec12045b20d42ddfb798e1a6ea06177c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, description=Ceph Grafana Container, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, vcs-type=git, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, release=236.1648460182) 2024-04-14T15:31:25.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[29517]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:31:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[29517]: Standby manager daemon x restarted 2024-04-14T15:31:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[29517]: Standby manager daemon x started 2024-04-14T15:31:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:31:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:31:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:31:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:31:25.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[24869]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:31:25.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[24869]: Standby manager daemon x restarted 2024-04-14T15:31:25.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[24869]: Standby manager daemon x started 2024-04-14T15:31:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:31:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:31:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:31:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:24 smithi016 ceph-mon[24869]: from='mgr.? 172.21.15.196:0/4286171791' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:31:25.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:24 smithi196 podman[55701]: 2024-04-14 15:31:24.881740793 +0000 UTC m=+0.393069293 container init aed2a23652fbe3f1e7674d4675a797aaec12045b20d42ddfb798e1a6ea06177c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, io.openshift.expose-services=, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , vcs-type=git, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182) 2024-04-14T15:31:25.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:24 smithi196 podman[55701]: 2024-04-14 15:31:24.884628683 +0000 UTC m=+0.395957184 container start aed2a23652fbe3f1e7674d4675a797aaec12045b20d42ddfb798e1a6ea06177c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, version=8.5, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.openshift.tags=base rhel8, name=ubi8, io.openshift.expose-services=, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, vcs-type=git, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, maintainer=Paul Cuzner , release=236.1648460182) 2024-04-14T15:31:25.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[55734]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:31:25.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:24 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[55734]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:31:25.166 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:24 smithi196 bash[55701]: aed2a23652fbe3f1e7674d4675a797aaec12045b20d42ddfb798e1a6ea06177c 2024-04-14T15:31:25.166 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:24 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:25.166 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:25 smithi196 podman[55761]: 2024-04-14 15:31:25.00004713 +0000 UTC m=+0.074114292 container died aed2a23652fbe3f1e7674d4675a797aaec12045b20d42ddfb798e1a6ea06177c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, vcs-type=git, description=Ceph Grafana Container, release=236.1648460182, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-14T15:31:25.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:25 smithi196 ceph-mon[25696]: mgrmap e33: y(active, since 30s), standbys: x 2024-04-14T15:31:25.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:25 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:31:25.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:25 smithi196 ceph-mon[25696]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:25.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:25 smithi196 podman[55761]: 2024-04-14 15:31:25.602170871 +0000 UTC m=+0.676238021 container remove aed2a23652fbe3f1e7674d4675a797aaec12045b20d42ddfb798e1a6ea06177c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, version=8.5, architecture=x86_64, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.component=ubi8-container, name=ubi8, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , description=Ceph Grafana Container) 2024-04-14T15:31:25.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:25 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:31:25.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:25 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:31:26.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:25 smithi016 ceph-mon[29517]: mgrmap e33: y(active, since 30s), standbys: x 2024-04-14T15:31:26.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:25 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:31:26.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:25 smithi016 ceph-mon[29517]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:26.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:25 smithi016 ceph-mon[24869]: mgrmap e33: y(active, since 30s), standbys: x 2024-04-14T15:31:26.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:25 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:31:26.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:25 smithi016 ceph-mon[24869]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:27.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-14T15:31:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-14T15:31:27.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-14T15:31:27.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-14T15:31:27.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T15:31:27.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:26 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-14T15:31:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:31:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:31:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-14T15:31:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-14T15:31:27.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-14T15:31:27.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-14T15:31:27.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T15:31:27.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:26 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-14T15:31:28.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:27 smithi196 ceph-mon[25696]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-04-14T15:31:28.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:27 smithi196 ceph-mon[25696]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:28.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:27 smithi196 ceph-mon[25696]: Upgrade: Setting container_image for all mgr 2024-04-14T15:31:28.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:27 smithi196 ceph-mon[25696]: Upgrade: It appears safe to stop mon.a 2024-04-14T15:31:28.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:27 smithi016 ceph-mon[24869]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-04-14T15:31:28.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:27 smithi016 ceph-mon[24869]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:28.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:27 smithi016 ceph-mon[24869]: Upgrade: Setting container_image for all mgr 2024-04-14T15:31:28.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:27 smithi016 ceph-mon[24869]: Upgrade: It appears safe to stop mon.a 2024-04-14T15:31:28.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:27 smithi016 ceph-mon[29517]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1010 B/s rd, 0 op/s 2024-04-14T15:31:28.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:27 smithi016 ceph-mon[29517]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:28.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:27 smithi016 ceph-mon[29517]: Upgrade: Setting container_image for all mgr 2024-04-14T15:31:28.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:27 smithi016 ceph-mon[29517]: Upgrade: It appears safe to stop mon.a 2024-04-14T15:31:28.914 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:28 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:31:29.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[29517]: Upgrade: Updating mon.a 2024-04-14T15:31:29.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:29.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:31:29.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:31:29.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:29.239 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[29517]: Deploying daemon mon.a on smithi016 2024-04-14T15:31:29.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[24869]: Upgrade: Updating mon.a 2024-04-14T15:31:29.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:29.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:31:29.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:31:29.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[24869]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:29.240 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:29 smithi016 ceph-mon[24869]: Deploying daemon mon.a on smithi016 2024-04-14T15:31:29.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:29 smithi196 ceph-mon[25696]: Upgrade: Updating mon.a 2024-04-14T15:31:29.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:29 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' 2024-04-14T15:31:29.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:29 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:31:29.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:29 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:31:29.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:29 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:29.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:29 smithi196 ceph-mon[25696]: Deploying daemon mon.a on smithi016 2024-04-14T15:31:30.096 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:30 smithi016 ceph-mon[29517]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:30.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:29 smithi016 systemd[1]: Stopping Ceph mon.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:30.097 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:30 smithi016 ceph-mon[24869]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:30.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a[24865]: 2024-04-14T15:31:30.093+0000 7fc7c5c5c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:31:30.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a[24865]: 2024-04-14T15:31:30.093+0000 7fc7c5c5c700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-14T15:31:30.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:30 smithi016 podman[64606]: 2024-04-14 15:31:30.125490394 +0000 UTC m=+0.143217751 container died e8cfd2da2fd965d6d8e193f500806c654d6a7c39f077d54f30dea8298fdf5308 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.tags=base centos centos-stream, name=centos-stream, release=754, GIT_BRANCH=HEAD, distribution-scope=public, ceph=True, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux ) 2024-04-14T15:31:30.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:30] ENGINE Bus STOPPING 2024-04-14T15:31:30.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:30] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:31:30.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:30] ENGINE Bus STOPPED 2024-04-14T15:31:30.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:30] ENGINE Bus STARTING 2024-04-14T15:31:30.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:30] ENGINE Serving on http://:::9283 2024-04-14T15:31:30.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:30] ENGINE Bus STARTED 2024-04-14T15:31:30.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:30 smithi196 ceph-mon[25696]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:30.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:30 smithi016 podman[64606]: 2024-04-14 15:31:30.511195812 +0000 UTC m=+0.528923168 container cleanup e8cfd2da2fd965d6d8e193f500806c654d6a7c39f077d54f30dea8298fdf5308 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., release=754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vcs-type=git, maintainer=Guillaume Abrioux ) 2024-04-14T15:31:30.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:30 smithi016 bash[64606]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a 2024-04-14T15:31:31.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:30 smithi016 podman[64632]: 2024-04-14 15:31:30.74571187 +0000 UTC m=+0.615486165 container remove e8cfd2da2fd965d6d8e193f500806c654d6a7c39f077d54f30dea8298fdf5308 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, ceph=True, com.redhat.component=centos-stream-container, version=8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , release=754) 2024-04-14T15:31:31.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:30 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-14T15:31:31.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:30 smithi016 systemd[1]: Stopped Ceph mon.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:31.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:30 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.a.service: Consumed 13.976s CPU time. 2024-04-14T15:31:31.087 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 systemd[1]: Starting Ceph mon.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:31.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 podman[64766]: 2024-04-14 15:31:31.297194329 +0000 UTC m=+0.110982848 container create 152c4b9fda2092b8d247f1295f7f190fc009ae33dfb9121dca73fe144e2203f6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-14T15:31:31.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 podman[64766]: 2024-04-14 15:31:31.203879225 +0000 UTC m=+0.017667700 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:31:31.723 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:31:31] "GET /metrics HTTP/1.1" 200 38233 "" "Prometheus/2.33.4" 2024-04-14T15:31:31.723 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 podman[64766]: 2024-04-14 15:31:31.575031203 +0000 UTC m=+0.388819676 container init 152c4b9fda2092b8d247f1295f7f190fc009ae33dfb9121dca73fe144e2203f6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, GIT_CLEAN=True) 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 podman[64766]: 2024-04-14 15:31:31.578372375 +0000 UTC m=+0.392160848 container start 152c4b9fda2092b8d247f1295f7f190fc009ae33dfb9121dca73fe144e2203f6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev), process ceph-mon, pid 2 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: pidfile_write: ignore empty --pid-file 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: load: jerasure load: lrc 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: RocksDB version: 7.9.2 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Git sha 0 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Compile date 2024-04-06 20:28:24 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: DB SUMMARY 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: DB Session ID: 068C8HXY2SA5H6SMV9NC 2024-04-14T15:31:31.724 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: CURRENT file: CURRENT 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: MANIFEST file: MANIFEST-000015 size: 980 Bytes 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3340647 ; 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.error_if_exists: 0 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.create_if_missing: 0 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.paranoid_checks: 1 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.env: 0x55ffe6477c60 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.fs: PosixFileSystem 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.info_log: 0x55ffe7ee7be0 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T15:31:31.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.statistics: (nil) 2024-04-14T15:31:31.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.use_fsync: 0 2024-04-14T15:31:31.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_log_file_size: 0 2024-04-14T15:31:31.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T15:31:31.726 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.allow_fallocate: 1 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.use_direct_reads: 0 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.db_log_dir: 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.wal_dir: 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T15:31:31.727 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.write_buffer_manager: 0x55ffe7eb8820 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T15:31:31.728 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.unordered_write: 0 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.row_cache: None 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.wal_filter: None 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.two_write_queues: 0 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.wal_compression: 0 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.atomic_flush: 0 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T15:31:31.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.log_readahead_size: 0 2024-04-14T15:31:31.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T15:31:31.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T15:31:31.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T15:31:31.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T15:31:31.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T15:31:31.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T15:31:31.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-14T15:31:31.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_background_jobs: 2 2024-04-14T15:31:31.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_background_compactions: -1 2024-04-14T15:31:31.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_subcompactions: 1 2024-04-14T15:31:31.732 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_open_files: -1 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_background_flushes: -1 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Compression algorithms supported: 2024-04-14T15:31:31.733 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: kZSTD supported: 0 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: kXpressCompression supported: 0 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: kZlibCompression supported: 1 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: kSnappyCompression supported: 1 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: kLZ4Compression supported: 1 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: kBZip2Compression supported: 0 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.merge_operator: 2024-04-14T15:31:31.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_filter: None 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ffe7ee7dc0) 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 4 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-04-14T15:31:31.735 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x55ffe7eeb090 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-04-14T15:31:31.736 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: bloomfilter 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 5 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: max_auto_readahead_size: 262144 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: prepopulate_block_cache: 0 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: initial_auto_readahead_size: 8192 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compression: NoCompression 2024-04-14T15:31:31.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.num_levels: 7 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:31:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T15:31:31.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T15:31:31.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T15:31:31.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T15:31:31.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T15:31:31.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T15:31:31.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T15:31:31.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T15:31:31.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T15:31:31.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T15:31:31.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T15:31:31.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T15:31:31.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-14T15:31:31.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.arena_block_size: 1048576 2024-04-14T15:31:31.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-14T15:31:31.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.inplace_update_support: 0 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.bloom_locality: 0 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.max_successive_merges: 0 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.ttl: 2592000 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-14T15:31:31.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.enable_blob_files: false 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.min_blob_size: 0 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.blob_file_starting_level: 0 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12152, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-04-14T15:31:31.744 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1a531674-f9b5-4d47-abbd-9b5d3b09e6c8 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108691607653, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108691616955, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3075396, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12153, "largest_seqno": 13613, "table_properties": {"data_size": 3068394, "index_size": 4357, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1605, "raw_key_size": 15050, "raw_average_key_size": 23, "raw_value_size": 3054511, "raw_average_value_size": 4802, "num_data_blocks": 198, "num_entries": 636, "num_filter_entries": 636, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713108691, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1a531674-f9b5-4d47-abbd-9b5d3b09e6c8", "db_session_id": "068C8HXY2SA5H6SMV9NC", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108691617064, "job": 1, "event": "recovery_finished"} 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ffe7fda000 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: DB pointer 0x55ffe7fc4000 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T15:31:31.745 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 1/0 2.93 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 621.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: L6 1/0 8.07 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 11.00 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 621.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 621.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:31:31.746 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 621.9 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 250.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 250.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:31:31.747 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache BinnedLRUCache@0x55ffe7eeb090#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: starting mon.a rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: mon.a@-1(???) e3 preinit fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: mon.a@-1(???).mds e1 new map 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: mon.a@-1(???).mds e1 print_map 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: mon.a@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-14T15:31:31.748 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T15:31:31.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T15:31:31.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T15:31:31.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 ceph-mon[64807]: mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-04-14T15:31:31.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 bash[64766]: 152c4b9fda2092b8d247f1295f7f190fc009ae33dfb9121dca73fe144e2203f6 2024-04-14T15:31:31.749 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:31 smithi016 systemd[1]: Started Ceph mon.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:32.406 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (88s) 33s ago 6m 15.0M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 unknown 11s ago 6m - - 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 unknown 33s ago 5m - - 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (15s) 11s ago 10m 67.2M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (52s) 33s ago 12m 474M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 starting - - - 2048M 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (11m) 11s ago 11m 42.4M 2048M 17.2.0 e1d6a67b021e f933c5fa37fd 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (11m) 33s ago 11m 43.9M 2048M 17.2.0 e1d6a67b021e 8629057bb8e9 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (98s) 33s ago 7m 9.99M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (70s) 11s ago 6m 16.9M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (10m) 33s ago 10m 52.4M 2737M 17.2.0 e1d6a67b021e b812f3e0fa81 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (10m) 33s ago 10m 51.9M 2737M 17.2.0 e1d6a67b021e c275d1a0eaeb 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (9m) 33s ago 9m 50.1M 2737M 17.2.0 e1d6a67b021e 592ce301a74a 2024-04-14T15:31:32.407 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (9m) 33s ago 9m 50.6M 2737M 17.2.0 e1d6a67b021e 1ed8730c3fc1 2024-04-14T15:31:32.408 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (9m) 11s ago 9m 53.8M 2992M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:31:32.408 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (8m) 11s ago 8m 50.5M 2992M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:31:32.408 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (8m) 11s ago 8m 48.8M 2992M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:31:32.408 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (7m) 11s ago 7m 53.9M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:31:32.408 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (21s) 11s ago 7m 49.8M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:31:32.408 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (5m) 33s ago 5m 73.3M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:31:32.408 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (5m) 11s ago 5m 74.0M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:31:32.408 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (5m) 33s ago 5m 71.4M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:31:32.408 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (5m) 11s ago 5m 74.2M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:31:32.820 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": {}, 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:31:32.821 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:31:32.822 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-04-14T15:31:32.822 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:31:32.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:31:32.822 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:31:33.166 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:31:33.167 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:31:33.167 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:31:33.167 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:31:33.167 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:31:33.167 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-14T15:31:33.167 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:31:33.167 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "2/23 daemons upgraded", 2024-04-14T15:31:33.167 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mon daemons", 2024-04-14T15:31:33.167 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:31:33.167 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:31:33.581 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-14T15:31:33.582 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-14T15:31:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: daemon iscsi.foo.smithi016.sijcrc on smithi016 is in unknown state 2024-04-14T15:31:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:31:33.914 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:31:35.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ignoring --setuser ceph since I am not root 2024-04-14T15:31:35.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ignoring --setgroup ceph since I am not root 2024-04-14T15:31:35.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mgr[60472]: -- 172.21.15.16:0/1284428023 <== mon.1 v2:172.21.15.16:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e9038c85a0 con 0x55e90389c800 2024-04-14T15:31:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:31:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:31:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:31:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 629 B/s rd, 0 op/s 2024-04-14T15:31:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: mon.a calling monitor election 2024-04-14T15:31:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/4264554952' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: from='client.35159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/2447997990' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: monmap epoch 3 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: last_changed 2024-04-14T15:20:26.737782+0000 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: created 2024-04-14T15:18:56.351711+0000 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: min_mon_release 17 (quincy) 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: election_strategy: 1 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:31:35.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: fsmap 2024-04-14T15:31:35.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T15:31:35.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: mgrmap e33: y(active, since 40s), standbys: x 2024-04-14T15:31:35.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-14T15:31:35.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-14T15:31:35.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: daemon iscsi.foo.smithi016.sijcrc on smithi016 is in unknown state 2024-04-14T15:31:35.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:31:35.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: from='mgr.15285 ' entity='' 2024-04-14T15:31:35.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[64807]: mgrmap e34: y(active, since 40s), standbys: x 2024-04-14T15:31:35.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:31:35.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:31:35.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 629 B/s rd, 0 op/s 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: mon.a calling monitor election 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/4264554952' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: from='client.35159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: from='client.? 172.21.15.16:0/2447997990' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: monmap epoch 3 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:31:35.114 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: last_changed 2024-04-14T15:20:26.737782+0000 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: created 2024-04-14T15:18:56.351711+0000 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: min_mon_release 17 (quincy) 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: election_strategy: 1 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: fsmap 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: mgrmap e33: y(active, since 40s), standbys: x 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: daemon iscsi.foo.smithi016.sijcrc on smithi016 is in unknown state 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: from='mgr.15285 ' entity='' 2024-04-14T15:31:35.115 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:34 smithi016 ceph-mon[29517]: mgrmap e34: y(active, since 40s), standbys: x 2024-04-14T15:31:35.165 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: ignoring --setuser ceph since I am not root 2024-04-14T15:31:35.165 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: ignoring --setgroup ceph since I am not root 2024-04-14T15:31:35.165 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mgr[54656]: -- 172.21.15.196:0/554621899 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56022db045a0 con 0x56022dad8c00 2024-04-14T15:31:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:31:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:31:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: from='mgr.15285 172.21.15.16:0/4153378431' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:31:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 629 B/s rd, 0 op/s 2024-04-14T15:31:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: mon.a calling monitor election 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/4264554952' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: from='client.35159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/2447997990' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: monmap epoch 3 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: last_changed 2024-04-14T15:20:26.737782+0000 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: created 2024-04-14T15:18:56.351711+0000 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: min_mon_release 17 (quincy) 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: election_strategy: 1 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-14T15:31:35.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:31:35.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:31:35.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: fsmap 2024-04-14T15:31:35.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T15:31:35.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: mgrmap e33: y(active, since 40s), standbys: x 2024-04-14T15:31:35.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-14T15:31:35.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-14T15:31:35.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: daemon iscsi.foo.smithi016.sijcrc on smithi016 is in unknown state 2024-04-14T15:31:35.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:31:35.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: from='mgr.15285 ' entity='' 2024-04-14T15:31:35.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:34 smithi196 ceph-mon[25696]: mgrmap e34: y(active, since 40s), standbys: x 2024-04-14T15:31:35.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:35.243+0000 7fda9fe95200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:31:35.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:35 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:35.340+0000 7fda9fe95200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:31:35.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:35.225+0000 7f69091df200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T15:31:35.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:35.321+0000 7f69091df200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T15:31:35.931 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:35 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-04-14T15:31:35.931 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:35 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:35.931 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:35 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:35.931 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:35 smithi196 podman[55927]: 2024-04-14 15:31:35.796378618 +0000 UTC m=+0.017235146 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:31:36.192 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:35 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:35.990+0000 7f69091df200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:31:36.192 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:36 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:36.085+0000 7f69091df200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:31:36.192 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:35 smithi196 podman[55927]: 2024-04-14 15:31:35.931595875 +0000 UTC m=+0.152452382 container create be76dc5d70c2fec9bd05c043e86b2328ba99ea9b2d36ee74465f9e82703a4b92 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, release=236.1648460182, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, distribution-scope=public, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, io.openshift.expose-services=) 2024-04-14T15:31:36.346 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:36 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:36.005+0000 7fda9fe95200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T15:31:36.346 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:36 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:36.101+0000 7fda9fe95200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T15:31:36.346 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:36 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:36.247+0000 7fda9fe95200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:31:36.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:36 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:36.345+0000 7fda9fe95200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:31:36.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:36 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:36.432+0000 7fda9fe95200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:31:36.630 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:36 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:36.241+0000 7f69091df200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T15:31:36.630 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:36 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:36.345+0000 7f69091df200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T15:31:36.631 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:36 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:36.429+0000 7f69091df200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T15:31:36.631 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:36 smithi196 podman[55927]: 2024-04-14 15:31:36.192830373 +0000 UTC m=+0.413686879 container init be76dc5d70c2fec9bd05c043e86b2328ba99ea9b2d36ee74465f9e82703a4b92 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, com.redhat.component=ubi8-container, description=Ceph Grafana Container, architecture=x86_64, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., distribution-scope=public, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.openshift.expose-services=) 2024-04-14T15:31:36.631 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:36 smithi196 podman[55927]: 2024-04-14 15:31:36.195889469 +0000 UTC m=+0.416745974 container start be76dc5d70c2fec9bd05c043e86b2328ba99ea9b2d36ee74465f9e82703a4b92 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, io.buildah.version=1.24.2, architecture=x86_64, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, description=Ceph Grafana Container, distribution-scope=public, name=ubi8, io.openshift.expose-services=) 2024-04-14T15:31:36.631 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:36 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[55943]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:31:36.631 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:36 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[55943]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:31:36.631 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:36 smithi196 bash[55927]: be76dc5d70c2fec9bd05c043e86b2328ba99ea9b2d36ee74465f9e82703a4b92 2024-04-14T15:31:36.631 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:36 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:36.631 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:36 smithi196 podman[55961]: 2024-04-14 15:31:36.332107137 +0000 UTC m=+0.101902943 container died be76dc5d70c2fec9bd05c043e86b2328ba99ea9b2d36ee74465f9e82703a4b92 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, architecture=x86_64, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, description=Ceph Grafana Container, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, io.openshift.expose-services=, version=8.5) 2024-04-14T15:31:36.903 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:36 smithi196 ceph-mon[25696]: mgrmap e35: y(active, since 41s), standbys: x 2024-04-14T15:31:36.903 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:36 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:36.703+0000 7f69091df200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:31:36.903 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:36 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:36.813+0000 7f69091df200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:31:36.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:36 smithi016 ceph-mon[64807]: mgrmap e35: y(active, since 41s), standbys: x 2024-04-14T15:31:36.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:36 smithi016 ceph-mon[29517]: mgrmap e35: y(active, since 41s), standbys: x 2024-04-14T15:31:36.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:36 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:36.706+0000 7fda9fe95200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T15:31:36.911 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:36 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:36.819+0000 7fda9fe95200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T15:31:37.165 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:36 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:36.902+0000 7f69091df200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:31:37.165 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:36.999+0000 7f69091df200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:31:37.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:36 smithi196 podman[55961]: 2024-04-14 15:31:36.939176159 +0000 UTC m=+0.708971949 container remove be76dc5d70c2fec9bd05c043e86b2328ba99ea9b2d36ee74465f9e82703a4b92 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc., version=8.5, io.buildah.version=1.24.2, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.openshift.tags=base rhel8, release=236.1648460182, com.redhat.component=ubi8-container, maintainer=Paul Cuzner ) 2024-04-14T15:31:37.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:36 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:31:37.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:36 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:31:37.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:36 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:36.909+0000 7fda9fe95200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T15:31:37.189 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:37 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:37.009+0000 7fda9fe95200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T15:31:37.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:37 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:37.188+0000 7fda9fe95200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:31:37.664 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:37 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:37.178+0000 7f69091df200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T15:31:38.414 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:38.137+0000 7f69091df200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:31:38.415 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:38.268+0000 7f69091df200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:31:38.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:38 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:38.187+0000 7fda9fe95200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T15:31:38.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:38 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:38.325+0000 7fda9fe95200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T15:31:38.897 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:38.606+0000 7f69091df200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:31:38.898 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:38.698+0000 7f69091df200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:31:38.898 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:31:38.974 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:38 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:38.672+0000 7fda9fe95200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T15:31:38.974 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:38 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:38.767+0000 7fda9fe95200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T15:31:38.974 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:38 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:38.973+0000 7fda9fe95200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:31:39.164 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:38.897+0000 7f69091df200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:31:39.165 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:39 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:39.086+0000 7f69091df200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:31:39.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:39 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:39.171+0000 7fda9fe95200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T15:31:39.643 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:39 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:39.379+0000 7f69091df200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:31:39.742 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:39 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:39.476+0000 7fda9fe95200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T15:31:39.914 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:39 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:39.643+0000 7f69091df200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:31:39.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:39 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:39.885+0000 7f69091df200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:31:39.993 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:39 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:39.741+0000 7fda9fe95200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T15:31:40.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:39 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:39.992+0000 7fda9fe95200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T15:31:41.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:40 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:40.838+0000 7fda9fe95200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:31:41.164 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:40 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:40.699+0000 7f69091df200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T15:31:41.605 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:41.277+0000 7f69091df200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:31:41.605 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:41.379+0000 7f69091df200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:31:41.773 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:41.430+0000 7fda9fe95200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T15:31:41.773 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:41.535+0000 7fda9fe95200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T15:31:41.774 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[29517]: Standby manager daemon x restarted 2024-04-14T15:31:41.774 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[29517]: Standby manager daemon x started 2024-04-14T15:31:41.774 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:31:41.774 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:31:41.774 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:31:41.774 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[29517]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:31:41.775 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[64807]: Standby manager daemon x restarted 2024-04-14T15:31:41.775 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[64807]: Standby manager daemon x started 2024-04-14T15:31:41.775 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[64807]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:31:41.775 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[64807]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:31:41.775 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[64807]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:31:41.775 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-mon[64807]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:31:41.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-mon[25696]: Standby manager daemon x restarted 2024-04-14T15:31:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-mon[25696]: Standby manager daemon x started 2024-04-14T15:31:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T15:31:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T15:31:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T15:31:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-mon[25696]: from='mgr.? 172.21.15.196:0/765011736' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T15:31:41.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:41.604+0000 7f69091df200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:31:41.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: [14/Apr/2024:15:31:41] ENGINE Bus STARTING 2024-04-14T15:31:41.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: CherryPy Checker: 2024-04-14T15:31:41.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: The Application mounted at '' has an empty config. 2024-04-14T15:31:41.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: 2024-04-14T15:31:41.915 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: [14/Apr/2024:15:31:41] ENGINE Serving on http://:::9283 2024-04-14T15:31:41.916 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:31:41 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x[54652]: [14/Apr/2024:15:31:41] ENGINE Bus STARTED 2024-04-14T15:31:42.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:41.773+0000 7fda9fe95200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T15:31:42.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:41] ENGINE Bus STARTING 2024-04-14T15:31:42.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: CherryPy Checker: 2024-04-14T15:31:42.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: The Application mounted at '' has an empty config. 2024-04-14T15:31:42.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:42.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:41] ENGINE Serving on http://:::9283 2024-04-14T15:31:42.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:41] ENGINE Bus STARTED 2024-04-14T15:31:42.995 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: mgrmap e36: y(active, since 47s), standbys: x 2024-04-14T15:31:42.995 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: Active manager daemon y restarted 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: Activating manager daemon y 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: mgrmap e37: y(active, starting, since 0.00673438s), standbys: x 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:31:42.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:31:42.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:31:42.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:31:42.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:31:42.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:31:42.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:31:42.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: Manager daemon y is now available 2024-04-14T15:31:42.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:31:42.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:31:42.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:31:42.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:31:42.997 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:42 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:31:43.022 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:31:42.797+0000 7fda5603a700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-14T15:31:43.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: mgrmap e36: y(active, since 47s), standbys: x 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: Active manager daemon y restarted 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: Activating manager daemon y 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: mgrmap e37: y(active, starting, since 0.00673438s), standbys: x 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:31:43.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: Manager daemon y is now available 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:31:43.024 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: mgrmap e36: y(active, since 47s), standbys: x 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: Active manager daemon y restarted 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: Activating manager daemon y 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: mgrmap e37: y(active, starting, since 0.00673438s), standbys: x 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:31:43.025 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: Manager daemon y is now available 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:31:43.026 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:42 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T15:31:43.846 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:43 smithi196 ceph-mon[25696]: [14/Apr/2024:15:31:42] ENGINE Bus STARTING 2024-04-14T15:31:43.846 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:43 smithi196 ceph-mon[25696]: [14/Apr/2024:15:31:42] ENGINE Serving on http://172.21.15.16:8765 2024-04-14T15:31:43.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:43 smithi196 ceph-mon[25696]: [14/Apr/2024:15:31:42] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:31:43.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:43 smithi196 ceph-mon[25696]: [14/Apr/2024:15:31:42] ENGINE Bus STARTED 2024-04-14T15:31:43.847 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:43 smithi196 ceph-mon[25696]: mgrmap e38: y(active, since 1.01053s), standbys: x 2024-04-14T15:31:43.847 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:31:44.042 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:43 smithi016 ceph-mon[29517]: [14/Apr/2024:15:31:42] ENGINE Bus STARTING 2024-04-14T15:31:44.042 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:43 smithi016 ceph-mon[29517]: [14/Apr/2024:15:31:42] ENGINE Serving on http://172.21.15.16:8765 2024-04-14T15:31:44.042 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:43 smithi016 ceph-mon[29517]: [14/Apr/2024:15:31:42] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:31:44.043 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:43 smithi016 ceph-mon[29517]: [14/Apr/2024:15:31:42] ENGINE Bus STARTED 2024-04-14T15:31:44.043 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:43 smithi016 ceph-mon[29517]: mgrmap e38: y(active, since 1.01053s), standbys: x 2024-04-14T15:31:44.043 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:43 smithi016 ceph-mon[64807]: [14/Apr/2024:15:31:42] ENGINE Bus STARTING 2024-04-14T15:31:44.043 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:43 smithi016 ceph-mon[64807]: [14/Apr/2024:15:31:42] ENGINE Serving on http://172.21.15.16:8765 2024-04-14T15:31:44.043 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:43 smithi016 ceph-mon[64807]: [14/Apr/2024:15:31:42] ENGINE Serving on https://172.21.15.16:7150 2024-04-14T15:31:44.043 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:43 smithi016 ceph-mon[64807]: [14/Apr/2024:15:31:42] ENGINE Bus STARTED 2024-04-14T15:31:44.043 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:43 smithi016 ceph-mon[64807]: mgrmap e38: y(active, since 1.01053s), standbys: x 2024-04-14T15:31:44.840 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:44 smithi196 ceph-mon[25696]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:44.841 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:44 smithi196 ceph-mon[25696]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:44.841 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:44 smithi196 ceph-mon[25696]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-04-14T15:31:44.841 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:44 smithi196 ceph-mon[25696]: Cluster is now healthy 2024-04-14T15:31:45.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:44 smithi016 ceph-mon[29517]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:45.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:44 smithi016 ceph-mon[29517]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:45.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:44 smithi016 ceph-mon[29517]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-04-14T15:31:45.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:44 smithi016 ceph-mon[29517]: Cluster is now healthy 2024-04-14T15:31:45.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:44 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:45.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:44 smithi016 ceph-mon[64807]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:45.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:44 smithi016 ceph-mon[64807]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-04-14T15:31:45.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:44 smithi016 ceph-mon[64807]: Cluster is now healthy 2024-04-14T15:31:46.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:45 smithi016 ceph-mon[29517]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:46.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:45 smithi016 ceph-mon[29517]: mgrmap e39: y(active, since 2s), standbys: x 2024-04-14T15:31:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:45 smithi016 ceph-mon[64807]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:45 smithi016 ceph-mon[64807]: mgrmap e39: y(active, since 2s), standbys: x 2024-04-14T15:31:46.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:45 smithi196 ceph-mon[25696]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:46.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:45 smithi196 ceph-mon[25696]: mgrmap e39: y(active, since 2s), standbys: x 2024-04-14T15:31:47.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 6. 2024-04-14T15:31:47.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:47.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:47.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:47 smithi196 ceph-mon[25696]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:47.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:47 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:47 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:47 smithi196 ceph-mon[25696]: mgrmap e40: y(active, since 4s), standbys: x 2024-04-14T15:31:47.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:47 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.398 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:47 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[29517]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:47.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[29517]: mgrmap e40: y(active, since 4s), standbys: x 2024-04-14T15:31:47.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[64807]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:31:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[64807]: mgrmap e40: y(active, since 4s), standbys: x 2024-04-14T15:31:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:47 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:47.664 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 podman[56874]: 2024-04-14 15:31:47.301433132 +0000 UTC m=+0.018878589 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:31:47.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 podman[56874]: 2024-04-14 15:31:47.447682039 +0000 UTC m=+0.165127529 container create ad6d56214a2ba7b003408784242478d46aa480f21a5d7a1ea072f566baa92ac2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, version=8.5, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, architecture=x86_64, vcs-type=git, release=236.1648460182, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-14T15:31:48.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 podman[56874]: 2024-04-14 15:31:47.717129613 +0000 UTC m=+0.434575061 container init ad6d56214a2ba7b003408784242478d46aa480f21a5d7a1ea072f566baa92ac2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, maintainer=Paul Cuzner , architecture=x86_64, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762) 2024-04-14T15:31:48.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 podman[56874]: 2024-04-14 15:31:47.720009225 +0000 UTC m=+0.437454673 container start ad6d56214a2ba7b003408784242478d46aa480f21a5d7a1ea072f566baa92ac2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, architecture=x86_64, maintainer=Paul Cuzner , distribution-scope=public, release=236.1648460182, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-14T15:31:48.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[56906]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:31:48.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[56906]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:31:48.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 bash[56874]: ad6d56214a2ba7b003408784242478d46aa480f21a5d7a1ea072f566baa92ac2 2024-04-14T15:31:48.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:48.079 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:47 smithi196 podman[56934]: 2024-04-14 15:31:47.808626864 +0000 UTC m=+0.051797540 container died ad6d56214a2ba7b003408784242478d46aa480f21a5d7a1ea072f566baa92ac2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, release=236.1648460182, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, io.openshift.expose-services=, name=ubi8, vendor=Red Hat, Inc., architecture=x86_64, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-14T15:31:48.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:48 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:48 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:48 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.362 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:48 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:48 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:48 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:48 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:48 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:48 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:48 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:48 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:48 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:48.664 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:48 smithi196 podman[56934]: 2024-04-14 15:31:48.3628602 +0000 UTC m=+0.606030855 container remove ad6d56214a2ba7b003408784242478d46aa480f21a5d7a1ea072f566baa92ac2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, description=Ceph Grafana Container, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , release=236.1648460182, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.buildah.version=1.24.2, vcs-type=git) 2024-04-14T15:31:48.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:48 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:31:48.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:48 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:31:49.486 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:49 smithi196 ceph-mon[25696]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-14T15:31:49.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:49 smithi016 ceph-mon[29517]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-14T15:31:49.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:49 smithi016 ceph-mon[64807]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-14T15:31:51.569 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:51 smithi016 ceph-mon[29517]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T15:31:51.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:51 smithi016 ceph-mon[64807]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T15:31:51.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:51 smithi196 ceph-mon[25696]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T15:31:51.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:51 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:31:51] "GET /metrics HTTP/1.1" 200 35450 "" "Prometheus/2.33.4" 2024-04-14T15:31:52.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:31:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[29517]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:31:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[29517]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:31:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[29517]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:52.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[29517]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:52.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:52.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:52.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:52.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:31:52.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[64807]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:31:52.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[64807]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:31:52.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[64807]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:52.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:52 smithi016 ceph-mon[64807]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:52 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:52 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:52 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:52 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:31:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:52 smithi196 ceph-mon[25696]: Updating smithi016:/etc/ceph/ceph.conf 2024-04-14T15:31:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:52 smithi196 ceph-mon[25696]: Updating smithi196:/etc/ceph/ceph.conf 2024-04-14T15:31:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:52 smithi196 ceph-mon[25696]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:52 smithi196 ceph-mon[25696]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T15:31:53.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T15:31:53.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:53.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:53.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:53.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:53.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T15:31:53.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:31:53.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:53.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T15:31:53.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-14T15:31:53.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[29517]: Upgrade: It appears safe to stop mon.c 2024-04-14T15:31:53.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T15:31:53.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:53.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:53.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:53.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:53.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T15:31:53.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.617 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:31:53.617 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:53.617 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T15:31:53.617 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-14T15:31:53.617 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:53 smithi016 ceph-mon[64807]: Upgrade: It appears safe to stop mon.c 2024-04-14T15:31:53.664 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:31:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T15:31:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-04-14T15:31:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: Updating smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: Updating smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T15:31:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T15:31:53.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:53.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:31:53.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:31:53.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T15:31:53.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-14T15:31:53.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:53 smithi196 ceph-mon[25696]: Upgrade: It appears safe to stop mon.c 2024-04-14T15:31:54.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[29517]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[29517]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:31:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[29517]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:31:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:31:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[29517]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[64807]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:31:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[64807]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:31:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:31:54.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:54 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:54 smithi196 ceph-mon[25696]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:31:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:54 smithi196 ceph-mon[25696]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:31:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:54 smithi196 ceph-mon[25696]: from='mgr.25330 ' entity='mgr.y' 2024-04-14T15:31:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:54 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:31:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:54 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:31:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:54 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:31:55.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:55 smithi196 ceph-mon[25696]: Upgrade: Updating mon.c 2024-04-14T15:31:55.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:55 smithi196 ceph-mon[25696]: Deploying daemon mon.c on smithi016 2024-04-14T15:31:55.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:31:55 smithi196 ceph-mon[25696]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T15:31:55.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:55 smithi016 ceph-mon[64807]: Upgrade: Updating mon.c 2024-04-14T15:31:55.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:55 smithi016 ceph-mon[64807]: Deploying daemon mon.c on smithi016 2024-04-14T15:31:55.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:31:55 smithi016 ceph-mon[64807]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T15:31:55.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:55 smithi016 ceph-mon[29517]: Upgrade: Updating mon.c 2024-04-14T15:31:55.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:55 smithi016 ceph-mon[29517]: Deploying daemon mon.c on smithi016 2024-04-14T15:31:55.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:55 smithi016 ceph-mon[29517]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T15:31:56.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:56 smithi016 systemd[1]: Stopping Ceph mon.c for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:56.742 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:56 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:56] ENGINE Bus STOPPING 2024-04-14T15:31:56.743 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:56 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:31:56.743 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:56 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:56] ENGINE Bus STOPPED 2024-04-14T15:31:56.743 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:56 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:56] ENGINE Bus STARTING 2024-04-14T15:31:56.743 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:56 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:56] ENGINE Serving on http://:::9283 2024-04-14T15:31:56.743 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:31:56 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:31:56] ENGINE Bus STARTED 2024-04-14T15:31:56.743 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:56 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c[29513]: 2024-04-14T15:31:56.407+0000 7fedcebeb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:31:56.743 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:56 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c[29513]: 2024-04-14T15:31:56.407+0000 7fedcebeb700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-14T15:31:56.743 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:56 smithi016 podman[67172]: 2024-04-14 15:31:56.469420272 +0000 UTC m=+0.290524091 container died 8629057bb8e95e9c24d1fd8fbcef970c0392c9580d56cccdf07d9a0ff1edf9d7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c, ceph=True, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vcs-type=git, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public) 2024-04-14T15:31:57.084 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:56 smithi016 podman[67172]: 2024-04-14 15:31:56.841381728 +0000 UTC m=+0.662485552 container cleanup 8629057bb8e95e9c24d1fd8fbcef970c0392c9580d56cccdf07d9a0ff1edf9d7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , architecture=x86_64, io.buildah.version=1.19.8, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, version=8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, release=754, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, ceph=True, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-14T15:31:57.084 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:56 smithi016 bash[67172]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c 2024-04-14T15:31:57.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 podman[67197]: 2024-04-14 15:31:57.084593576 +0000 UTC m=+0.614642685 container remove 8629057bb8e95e9c24d1fd8fbcef970c0392c9580d56cccdf07d9a0ff1edf9d7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c, io.openshift.expose-services=, RELEASE=HEAD, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-type=git, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., version=8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-14T15:31:57.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-14T15:31:57.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 systemd[1]: Stopped Ceph mon.c for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:57.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.c.service: Consumed 8.237s CPU time. 2024-04-14T15:31:57.760 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 systemd[1]: Starting Ceph mon.c for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:57.761 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 podman[67329]: 2024-04-14 15:31:57.635862815 +0000 UTC m=+0.114964321 container create da68d9a2587cb3f6d20190d467d264d33be530f3bd9652248f90850fa32eb278 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-04-14T15:31:57.761 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 podman[67329]: 2024-04-14 15:31:57.539120938 +0000 UTC m=+0.018222447 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:31:58.014 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 podman[67329]: 2024-04-14 15:31:57.89722757 +0000 UTC m=+0.376329070 container init da68d9a2587cb3f6d20190d467d264d33be530f3bd9652248f90850fa32eb278 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403) 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 podman[67329]: 2024-04-14 15:31:57.900406794 +0000 UTC m=+0.379508297 container start da68d9a2587cb3f6d20190d467d264d33be530f3bd9652248f90850fa32eb278 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev), process ceph-mon, pid 2 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: pidfile_write: ignore empty --pid-file 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: load: jerasure load: lrc 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: RocksDB version: 7.9.2 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Git sha 0 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Compile date 2024-04-06 20:28:24 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: DB SUMMARY 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: DB Session ID: 8BY096PXSHHN2LQS8JLW 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: CURRENT file: CURRENT 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-04-14T15:31:58.015 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 1060827 ; 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.error_if_exists: 0 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.create_if_missing: 0 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.paranoid_checks: 1 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.env: 0x5621039e5c60 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.fs: PosixFileSystem 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.info_log: 0x5621042b1be0 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.statistics: (nil) 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.use_fsync: 0 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_log_file_size: 0 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T15:31:58.016 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.allow_fallocate: 1 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.use_direct_reads: 0 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.db_log_dir: 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.wal_dir: 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T15:31:58.017 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.write_buffer_manager: 0x562104282820 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.unordered_write: 0 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T15:31:58.018 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.row_cache: None 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.wal_filter: None 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.two_write_queues: 0 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.wal_compression: 0 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.atomic_flush: 0 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.log_readahead_size: 0 2024-04-14T15:31:58.019 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_background_jobs: 2 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_background_compactions: -1 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_subcompactions: 1 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T15:31:58.020 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T15:31:58.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T15:31:58.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_open_files: -1 2024-04-14T15:31:58.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T15:31:58.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_background_flushes: -1 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Compression algorithms supported: 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: kZSTD supported: 0 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: kXpressCompression supported: 0 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: kZlibCompression supported: 1 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: kSnappyCompression supported: 1 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: kLZ4Compression supported: 1 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: kBZip2Compression supported: 0 2024-04-14T15:31:58.022 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.merge_operator: 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_filter: None 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5621042b1dc0) 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T15:31:58.023 INFO:journalctl@ceph.mon.c.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-04-14T15:31:58.024 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_type: 0 2024-04-14T15:31:58.024 INFO:journalctl@ceph.mon.c.smithi016.stdout: data_block_index_type: 0 2024-04-14T15:31:58.024 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_shortening: 1 2024-04-14T15:31:58.024 INFO:journalctl@ceph.mon.c.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T15:31:58.024 INFO:journalctl@ceph.mon.c.smithi016.stdout: checksum: 4 2024-04-14T15:31:58.024 INFO:journalctl@ceph.mon.c.smithi016.stdout: no_block_cache: 0 2024-04-14T15:31:58.024 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache: 0x5621042b5090 2024-04-14T15:31:58.024 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_options: 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: capacity : 536870912 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: num_shard_bits : 4 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: strict_capacity_limit : 0 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_compressed: (nil) 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: persistent_cache: (nil) 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_size: 4096 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_size_deviation: 10 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_restart_interval: 16 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_block_restart_interval: 1 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: metadata_block_size: 4096 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: partition_filters: 0 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: use_delta_encoding: 1 2024-04-14T15:31:58.025 INFO:journalctl@ceph.mon.c.smithi016.stdout: filter_policy: bloomfilter 2024-04-14T15:31:58.026 INFO:journalctl@ceph.mon.c.smithi016.stdout: whole_key_filtering: 1 2024-04-14T15:31:58.026 INFO:journalctl@ceph.mon.c.smithi016.stdout: verify_compression: 0 2024-04-14T15:31:58.026 INFO:journalctl@ceph.mon.c.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-04-14T15:31:58.026 INFO:journalctl@ceph.mon.c.smithi016.stdout: format_version: 5 2024-04-14T15:31:58.026 INFO:journalctl@ceph.mon.c.smithi016.stdout: enable_index_compression: 1 2024-04-14T15:31:58.026 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_align: 0 2024-04-14T15:31:58.026 INFO:journalctl@ceph.mon.c.smithi016.stdout: max_auto_readahead_size: 262144 2024-04-14T15:31:58.026 INFO:journalctl@ceph.mon.c.smithi016.stdout: prepopulate_block_cache: 0 2024-04-14T15:31:58.026 INFO:journalctl@ceph.mon.c.smithi016.stdout: initial_auto_readahead_size: 8192 2024-04-14T15:31:58.026 INFO:journalctl@ceph.mon.c.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-04-14T15:31:58.026 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T15:31:58.027 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T15:31:58.027 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compression: NoCompression 2024-04-14T15:31:58.027 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T15:31:58.027 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T15:31:58.027 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T15:31:58.027 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.num_levels: 7 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T15:31:58.028 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T15:31:58.029 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T15:31:58.030 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T15:31:58.030 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T15:31:58.030 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T15:31:58.030 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T15:31:58.030 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T15:31:58.030 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T15:31:58.030 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T15:31:58.030 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T15:31:58.030 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T15:31:58.030 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T15:31:58.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-14T15:31:58.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.arena_block_size: 1048576 2024-04-14T15:31:58.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T15:31:58.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T15:31:58.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T15:31:58.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T15:31:58.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T15:31:58.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T15:31:58.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T15:31:58.031 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.inplace_update_support: 0 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.bloom_locality: 0 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.max_successive_merges: 0 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T15:31:58.032 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.ttl: 2592000 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.enable_blob_files: false 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.min_blob_size: 0 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-14T15:31:58.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.blob_file_starting_level: 0 2024-04-14T15:31:58.034 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-14T15:31:58.034 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-14T15:31:58.034 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T15:31:58.034 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14986, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T15:31:58.034 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-14T15:31:58.034 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 851c5149-c7e0-4c0b-871d-9137b08fa5ae 2024-04-14T15:31:58.034 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108717929814, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-14T15:31:58.034 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-14T15:31:58.034 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108717933376, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 650122, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14991, "largest_seqno": 15261, "table_properties": {"data_size": 647818, "index_size": 946, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 325, "raw_key_size": 2967, "raw_average_key_size": 24, "raw_value_size": 644858, "raw_average_value_size": 5242, "num_data_blocks": 42, "num_entries": 123, "num_filter_entries": 123, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713108717, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "851c5149-c7e0-4c0b-871d-9137b08fa5ae", "db_session_id": "8BY096PXSHHN2LQS8JLW", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-14T15:31:58.034 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108717933469, "job": 1, "event": "recovery_finished"} 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5621043a4000 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: DB pointer 0x56210438e000 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** DB Stats ** 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:31:58.035 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: L0 1/0 634.88 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 317.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: L6 1/0 11.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: Sum 2/0 12.05 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 317.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 317.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:31:58.036 INFO:journalctl@ceph.mon.c.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 317.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative compaction: 0.00 GB write, 107.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval compaction: 0.00 GB write, 107.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: Block cache BinnedLRUCache@0x5621042b5090#2 capacity: 512.00 MB usage: 1.45 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-14T15:31:58.037 INFO:journalctl@ceph.mon.c.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.38 KB,7.15256e-05%) IndexBlock(1,1.08 KB,0.000205636%) Misc(1,0.00 KB,0%) 2024-04-14T15:31:58.038 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:31:58.038 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:31:58.038 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: starting mon.c rank 1 at public addrs [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] at bind addrs [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:31:58.038 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: mon.c@-1(???) e3 preinit fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:31:58.038 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: mon.c@-1(???).mds e1 new map 2024-04-14T15:31:58.038 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: mon.c@-1(???).mds e1 print_map 2024-04-14T15:31:58.038 INFO:journalctl@ceph.mon.c.smithi016.stdout: e1 2024-04-14T15:31:58.038 INFO:journalctl@ceph.mon.c.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T15:31:58.038 INFO:journalctl@ceph.mon.c.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T15:31:58.038 INFO:journalctl@ceph.mon.c.smithi016.stdout: legacy client fscid: -1 2024-04-14T15:31:58.039 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-04-14T15:31:58.039 INFO:journalctl@ceph.mon.c.smithi016.stdout: No filesystems configured 2024-04-14T15:31:58.039 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: mon.c@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-14T15:31:58.039 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T15:31:58.039 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T15:31:58.039 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T15:31:58.039 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:57 smithi016 ceph-mon[67348]: mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-04-14T15:31:58.039 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:58 smithi016 bash[67329]: da68d9a2587cb3f6d20190d467d264d33be530f3bd9652248f90850fa32eb278 2024-04-14T15:31:58.039 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:31:58 smithi016 systemd[1]: Started Ceph mon.c for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:58.688 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:58 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 7. 2024-04-14T15:31:58.688 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:58 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:58.688 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:58 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:31:58.688 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:58 smithi196 podman[58007]: 2024-04-14 15:31:58.557435592 +0000 UTC m=+0.017871984 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:31:58.688 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:31:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:31:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:31:58.938 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:58 smithi196 podman[58007]: 2024-04-14 15:31:58.688471383 +0000 UTC m=+0.148907757 container create 4a31c1af206994ac449e397aaf6b8bbab287acdadfb89b36a1720ac806677540 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, distribution-scope=public, version=8.5, io.openshift.expose-services=, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, release=236.1648460182, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container) 2024-04-14T15:31:58.938 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:58 smithi196 podman[58007]: 2024-04-14 15:31:58.908149708 +0000 UTC m=+0.368586082 container init 4a31c1af206994ac449e397aaf6b8bbab287acdadfb89b36a1720ac806677540 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, maintainer=Paul Cuzner , release=236.1648460182, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.openshift.expose-services=, vendor=Red Hat, Inc., description=Ceph Grafana Container) 2024-04-14T15:31:58.939 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:58 smithi196 podman[58007]: 2024-04-14 15:31:58.910925373 +0000 UTC m=+0.371361751 container start 4a31c1af206994ac449e397aaf6b8bbab287acdadfb89b36a1720ac806677540 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., vcs-type=git, distribution-scope=public, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, release=236.1648460182, maintainer=Paul Cuzner , name=ubi8, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-04-14T15:31:58.939 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[58022]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:31:58.939 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[58022]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:31:59.314 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:59 smithi196 bash[58007]: 4a31c1af206994ac449e397aaf6b8bbab287acdadfb89b36a1720ac806677540 2024-04-14T15:31:59.314 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:59 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:31:59.314 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:59 smithi196 podman[58038]: 2024-04-14 15:31:59.042028161 +0000 UTC m=+0.097196821 container died 4a31c1af206994ac449e397aaf6b8bbab287acdadfb89b36a1720ac806677540 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, description=Ceph Grafana Container, version=8.5, io.openshift.tags=base rhel8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, distribution-scope=public, io.openshift.expose-services=, name=ubi8) 2024-04-14T15:31:59.850 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:59 smithi196 podman[58038]: 2024-04-14 15:31:59.595000666 +0000 UTC m=+0.650169334 container remove 4a31c1af206994ac449e397aaf6b8bbab287acdadfb89b36a1720ac806677540 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , release=236.1648460182, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, vcs-type=git, io.openshift.tags=base rhel8, io.openshift.expose-services=) 2024-04-14T15:31:59.850 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:59 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:31:59.850 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:31:59 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:32:00.734 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:00.734 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: mon.c calling monitor election 2024-04-14T15:32:00.734 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:32:00.734 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: mon.a calling monitor election 2024-04-14T15:32:00.734 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:32:00.734 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: monmap epoch 3 2024-04-14T15:32:00.734 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:32:00.734 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: last_changed 2024-04-14T15:20:26.737782+0000 2024-04-14T15:32:00.734 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: created 2024-04-14T15:18:56.351711+0000 2024-04-14T15:32:00.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: min_mon_release 17 (quincy) 2024-04-14T15:32:00.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: election_strategy: 1 2024-04-14T15:32:00.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-14T15:32:00.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:32:00.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:32:00.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: fsmap 2024-04-14T15:32:00.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:32:00.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: mgrmap e40: y(active, since 17s), standbys: x 2024-04-14T15:32:00.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:00.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:00.736 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:00.736 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:00.736 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:00.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:00.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: mon.c calling monitor election 2024-04-14T15:32:00.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: mon.a calling monitor election 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: monmap epoch 3 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: last_changed 2024-04-14T15:20:26.737782+0000 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: created 2024-04-14T15:18:56.351711+0000 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: min_mon_release 17 (quincy) 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: election_strategy: 1 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: fsmap 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:32:00.737 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: mgrmap e40: y(active, since 17s), standbys: x 2024-04-14T15:32:00.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:00.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:00.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:00.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:00.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: mon.c calling monitor election 2024-04-14T15:32:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:32:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: mon.a calling monitor election 2024-04-14T15:32:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:32:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: monmap epoch 3 2024-04-14T15:32:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:32:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: last_changed 2024-04-14T15:20:26.737782+0000 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: created 2024-04-14T15:18:56.351711+0000 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: min_mon_release 17 (quincy) 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: election_strategy: 1 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: fsmap 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: mgrmap e40: y(active, since 17s), standbys: x 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:00 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:01.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:01 smithi016 ceph-mon[67348]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-14T15:32:01.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:01 smithi016 ceph-mon[64807]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-14T15:32:01.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:32:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:32:01] "GET /metrics HTTP/1.1" 200 38252 "" "Prometheus/2.33.4" 2024-04-14T15:32:01.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:01 smithi196 ceph-mon[25696]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-14T15:32:03.097 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:32:02 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:32:02.940+0000 7fda5603a700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-14T15:32:03.740 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:03 smithi016 ceph-mon[67348]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-14T15:32:03.741 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:03 smithi016 ceph-mon[64807]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-14T15:32:03.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:03 smithi196 ceph-mon[25696]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-14T15:32:03.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:03.951 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:32:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:04 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:04 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:04 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:04.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:04 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:04.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:04 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:04.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:04 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:04.651 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:32:04.651 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (2m) 0s ago 7m 17.7M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:32:04.651 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 unknown 18s ago 6m - - 2024-04-14T15:32:04.651 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (60s) 0s ago 5m 49.6M - 3.5 e1d6a67b021e 27d95fa4403f 2024-04-14T15:32:04.652 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (48s) 18s ago 11m 448M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:32:04.652 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (84s) 0s ago 12m 494M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:32:04.652 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (33s) 0s ago 13m 39.1M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:32:04.652 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (11m) 18s ago 11m 48.8M 2048M 17.2.0 e1d6a67b021e f933c5fa37fd 2024-04-14T15:32:04.652 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (7s) 0s ago 11m 17.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:32:04.652 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (2m) 0s ago 7m 17.5M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:32:04.653 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (103s) 18s ago 7m 17.2M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:32:04.653 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (11m) 0s ago 11m 53.5M 2737M 17.2.0 e1d6a67b021e b812f3e0fa81 2024-04-14T15:32:04.653 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (10m) 0s ago 10m 52.8M 2737M 17.2.0 e1d6a67b021e c275d1a0eaeb 2024-04-14T15:32:04.653 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (10m) 0s ago 10m 51.0M 2737M 17.2.0 e1d6a67b021e 592ce301a74a 2024-04-14T15:32:04.653 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (9m) 0s ago 9m 51.4M 2737M 17.2.0 e1d6a67b021e 1ed8730c3fc1 2024-04-14T15:32:04.653 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (9m) 18s ago 9m 54.0M 2992M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:32:04.653 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (9m) 18s ago 9m 50.5M 2992M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:32:04.653 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (8m) 18s ago 8m 49.0M 2992M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:32:04.654 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (8m) 18s ago 8m 54.1M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:32:04.654 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (53s) 18s ago 7m 60.2M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:32:04.654 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (6m) 0s ago 6m 74.7M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:32:04.654 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (6m) 18s ago 6m 74.5M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:32:04.654 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (6m) 0s ago 6m 72.6M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:32:04.654 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (5m) 18s ago 5m 74.9M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:32:04.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:04 smithi196 ceph-mon[25696]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:04 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:04 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:05.074 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:32:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:32:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T15:32:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:32:05.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 4 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:32:05.075 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:32:05.428 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:32:05.429 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:32:05.429 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:32:05.429 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:32:05.429 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:32:05.429 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-14T15:32:05.429 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:32:05.429 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "4/23 daemons upgraded", 2024-04-14T15:32:05.430 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mon daemons", 2024-04-14T15:32:05.430 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:32:05.430 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:32:05.848 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:05.848 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:05.848 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:05.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[67348]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:05.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[67348]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:05.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[67348]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:05.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[67348]: from='client.54115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:05.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/3953346603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:32:05.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:05.850 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:05.850 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[64807]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:05.850 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[64807]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:05.850 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[64807]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:05.850 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[64807]: from='client.54115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:05.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/3953346603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:32:05.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:05.851 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:05 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:05 smithi196 ceph-mon[25696]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:05 smithi196 ceph-mon[25696]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:05 smithi196 ceph-mon[25696]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:05 smithi196 ceph-mon[25696]: from='client.54115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:05 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/3953346603' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:32:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:05 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:05.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:05 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:06.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:06 smithi016 ceph-mon[67348]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:06.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:06 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/409075500' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:32:06.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:06 smithi016 ceph-mon[64807]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:06.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:06 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/409075500' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:32:06.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:06 smithi196 ceph-mon[25696]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:06.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:06 smithi196 ceph-mon[25696]: from='client.? 172.21.15.16:0/409075500' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:32:07.702 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:07 smithi016 ceph-mon[64807]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:07.703 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:07 smithi016 ceph-mon[67348]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:07.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:07 smithi196 ceph-mon[25696]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:08.914 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:09.644 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:09 smithi196 ceph-mon[25696]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:09.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:09 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:09.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:09 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:09.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:09 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:09.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:09 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:32:09.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:09 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:09.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:09 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:32:09.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:09 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:32:09.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:09 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T15:32:09.645 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:09 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-14T15:32:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[64807]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:32:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:32:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:32:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T15:32:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-14T15:32:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[67348]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:32:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:32:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:32:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T15:32:09.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:09 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-14T15:32:09.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:09 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 8. 2024-04-14T15:32:09.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:09 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:09.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:09 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:32:10.601 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:10 smithi196 ceph-mon[25696]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-04-14T15:32:10.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:10 smithi196 ceph-mon[25696]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:10.602 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:10 smithi196 ceph-mon[25696]: Upgrade: It appears safe to stop mon.b 2024-04-14T15:32:10.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:10 smithi016 ceph-mon[67348]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-04-14T15:32:10.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:10 smithi016 ceph-mon[67348]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:10.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:10 smithi016 ceph-mon[67348]: Upgrade: It appears safe to stop mon.b 2024-04-14T15:32:10.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:10 smithi016 ceph-mon[64807]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-04-14T15:32:10.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:10 smithi016 ceph-mon[64807]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:10.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:10 smithi016 ceph-mon[64807]: Upgrade: It appears safe to stop mon.b 2024-04-14T15:32:10.914 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:10 smithi196 podman[58276]: 2024-04-14 15:32:10.695883163 +0000 UTC m=+0.018325148 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:32:10.953 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:10 smithi196 podman[58276]: 2024-04-14 15:32:10.844915225 +0000 UTC m=+0.167357208 container create ccdef1dc3e334069b3945180a9afe9a67cbf46847d71d6fb9196880760f89cc0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, name=ubi8, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-14T15:32:11.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:11 smithi196 podman[58276]: 2024-04-14 15:32:11.097602007 +0000 UTC m=+0.420044003 container init ccdef1dc3e334069b3945180a9afe9a67cbf46847d71d6fb9196880760f89cc0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, name=ubi8, io.buildah.version=1.24.2, maintainer=Paul Cuzner , distribution-scope=public, io.openshift.expose-services=, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, version=8.5, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc.) 2024-04-14T15:32:11.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:11 smithi196 podman[58276]: 2024-04-14 15:32:11.10093599 +0000 UTC m=+0.423377973 container start ccdef1dc3e334069b3945180a9afe9a67cbf46847d71d6fb9196880760f89cc0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, vendor=Red Hat, Inc., distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.buildah.version=1.24.2, description=Ceph Grafana Container, release=236.1648460182, architecture=x86_64, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, name=ubi8) 2024-04-14T15:32:11.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[58356]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:32:11.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[58356]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:32:11.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:11 smithi196 bash[58276]: ccdef1dc3e334069b3945180a9afe9a67cbf46847d71d6fb9196880760f89cc0 2024-04-14T15:32:11.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:11 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:11.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:11 smithi196 podman[58385]: 2024-04-14 15:32:11.165370179 +0000 UTC m=+0.027824755 container died ccdef1dc3e334069b3945180a9afe9a67cbf46847d71d6fb9196880760f89cc0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, architecture=x86_64, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.openshift.expose-services=, io.openshift.tags=base rhel8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, distribution-scope=public) 2024-04-14T15:32:11.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:32:11] "GET /metrics HTTP/1.1" 200 38252 "" "Prometheus/2.33.4" 2024-04-14T15:32:11.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[67348]: Upgrade: Updating mon.b 2024-04-14T15:32:11.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:11.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:11.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:11.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:11.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[67348]: Deploying daemon mon.b on smithi196 2024-04-14T15:32:11.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[64807]: Upgrade: Updating mon.b 2024-04-14T15:32:11.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:11.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:11.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:11.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:11.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:11 smithi016 ceph-mon[64807]: Deploying daemon mon.b on smithi196 2024-04-14T15:32:11.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:11 smithi196 ceph-mon[25696]: Upgrade: Updating mon.b 2024-04-14T15:32:11.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:11 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:11.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:11 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:11.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:11 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:11.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:11 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:11.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:11 smithi196 ceph-mon[25696]: Deploying daemon mon.b on smithi196 2024-04-14T15:32:11.993 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:11 smithi196 podman[58385]: 2024-04-14 15:32:11.734956282 +0000 UTC m=+0.597410900 container remove ccdef1dc3e334069b3945180a9afe9a67cbf46847d71d6fb9196880760f89cc0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, vcs-type=git, version=8.5, architecture=x86_64, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.buildah.version=1.24.2, io.openshift.tags=base rhel8, distribution-scope=public) 2024-04-14T15:32:11.993 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:11 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:32:11.993 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:11 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:32:12.933 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:12 smithi196 ceph-mon[25696]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:12.934 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:12 smithi196 ceph-mon[25696]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:13.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:12 smithi016 ceph-mon[67348]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:13.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:13.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:12 smithi016 ceph-mon[64807]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:13.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:13.361 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:13 smithi196 systemd[1]: Stopping Ceph mon.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:32:13.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:13.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b[25681]: 2024-04-14T15:32:13.387+0000 7fa0991c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:32:13.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b[25681]: 2024-04-14T15:32:13.387+0000 7fa0991c1700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-14T15:32:13.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:13 smithi196 podman[58504]: 2024-04-14 15:32:13.551159554 +0000 UTC m=+0.271838873 container died f933c5fa37fdc610b50da82a90846d439156675daec5bd0ded89ed51a550192b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, architecture=x86_64, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, version=8, GIT_CLEAN=True, ceph=True, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0) 2024-04-14T15:32:14.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:13 smithi196 podman[58504]: 2024-04-14 15:32:13.877536624 +0000 UTC m=+0.598215935 container cleanup f933c5fa37fdc610b50da82a90846d439156675daec5bd0ded89ed51a550192b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.openshift.expose-services=, ceph=True, version=8, io.buildah.version=1.19.8, release=754, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-14T15:32:14.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:13 smithi196 bash[58504]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b 2024-04-14T15:32:14.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:14 smithi196 podman[58527]: 2024-04-14 15:32:14.053676621 +0000 UTC m=+0.498454857 container remove f933c5fa37fdc610b50da82a90846d439156675daec5bd0ded89ed51a550192b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.openshift.expose-services=, release=754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, version=8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-14T15:32:14.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:14 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-14T15:32:14.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:14 smithi196 systemd[1]: Stopped Ceph mon.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:14.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:14 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.b.service: Consumed 9.372s CPU time. 2024-04-14T15:32:14.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:14 smithi196 systemd[1]: Starting Ceph mon.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:32:14.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:14 smithi196 podman[58660]: 2024-04-14 15:32:14.626256317 +0000 UTC m=+0.018209555 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:32:14.984 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:14 smithi196 podman[58660]: 2024-04-14 15:32:14.746831338 +0000 UTC m=+0.138784563 container create 72817f80f666c7de49b830bec27e1ed08cf6ec896a527e08b538ec376a141bf4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:32:15.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:14 smithi196 podman[58660]: 2024-04-14 15:32:14.983363005 +0000 UTC m=+0.375316235 container init 72817f80f666c7de49b830bec27e1ed08cf6ec896a527e08b538ec376a141bf4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5) 2024-04-14T15:32:15.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:14 smithi196 podman[58660]: 2024-04-14 15:32:14.987508278 +0000 UTC m=+0.379461512 container start 72817f80f666c7de49b830bec27e1ed08cf6ec896a527e08b538ec376a141bf4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:32:15.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T15:32:15.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev), process ceph-mon, pid 2 2024-04-14T15:32:15.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: pidfile_write: ignore empty --pid-file 2024-04-14T15:32:15.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: load: jerasure load: lrc 2024-04-14T15:32:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: RocksDB version: 7.9.2 2024-04-14T15:32:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Git sha 0 2024-04-14T15:32:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Compile date 2024-04-06 20:28:24 2024-04-14T15:32:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: DB SUMMARY 2024-04-14T15:32:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: DB Session ID: 4KVHZ1AZPC9NX6YEDV5L 2024-04-14T15:32:15.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: CURRENT file: CURRENT 2024-04-14T15:32:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T15:32:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-04-14T15:32:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-04-14T15:32:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1807831 ; 2024-04-14T15:32:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.error_if_exists: 0 2024-04-14T15:32:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.create_if_missing: 0 2024-04-14T15:32:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.paranoid_checks: 1 2024-04-14T15:32:15.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-14T15:32:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T15:32:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-14T15:32:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.env: 0x56197221dc60 2024-04-14T15:32:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.fs: PosixFileSystem 2024-04-14T15:32:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.info_log: 0x5619734a1be0 2024-04-14T15:32:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T15:32:15.419 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.statistics: (nil) 2024-04-14T15:32:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.use_fsync: 0 2024-04-14T15:32:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_log_file_size: 0 2024-04-14T15:32:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T15:32:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T15:32:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T15:32:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T15:32:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.allow_fallocate: 1 2024-04-14T15:32:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T15:32:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T15:32:15.422 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.use_direct_reads: 0 2024-04-14T15:32:15.423 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T15:32:15.423 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T15:32:15.423 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.db_log_dir: 2024-04-14T15:32:15.423 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.wal_dir: 2024-04-14T15:32:15.423 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T15:32:15.423 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T15:32:15.423 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.write_buffer_manager: 0x561973472820 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T15:32:15.424 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.unordered_write: 0 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.row_cache: None 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.wal_filter: None 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.two_write_queues: 0 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.wal_compression: 0 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.atomic_flush: 0 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T15:32:15.425 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.log_readahead_size: 0 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_background_jobs: 2 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_background_compactions: -1 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_subcompactions: 1 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T15:32:15.426 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T15:32:15.427 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T15:32:15.427 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T15:32:15.427 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T15:32:15.427 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T15:32:15.427 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T15:32:15.427 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T15:32:15.427 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_open_files: -1 2024-04-14T15:32:15.427 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T15:32:15.427 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T15:32:15.427 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_background_flushes: -1 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Compression algorithms supported: 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: kZSTD supported: 0 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: kXpressCompression supported: 0 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: kZlibCompression supported: 1 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: kSnappyCompression supported: 1 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: kLZ4Compression supported: 1 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: kBZip2Compression supported: 0 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-14T15:32:15.428 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.merge_operator: 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_filter: None 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5619734a1dc0) 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks: 1 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_top_level_index_and_filter: 1 2024-04-14T15:32:15.429 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_type: 0 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_index_type: 0 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_shortening: 1 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: checksum: 4 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: no_block_cache: 0 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache: 0x5619734a5090 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_name: BinnedLRUCache 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_options: 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: capacity : 536870912 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_shard_bits : 4 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: strict_capacity_limit : 0 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: high_pri_pool_ratio: 0.000 2024-04-14T15:32:15.430 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_compressed: (nil) 2024-04-14T15:32:15.431 INFO:journalctl@ceph.mon.b.smithi196.stdout: persistent_cache: (nil) 2024-04-14T15:32:15.431 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size: 4096 2024-04-14T15:32:15.431 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size_deviation: 10 2024-04-14T15:32:15.431 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_restart_interval: 16 2024-04-14T15:32:15.431 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_block_restart_interval: 1 2024-04-14T15:32:15.431 INFO:journalctl@ceph.mon.b.smithi196.stdout: metadata_block_size: 4096 2024-04-14T15:32:15.431 INFO:journalctl@ceph.mon.b.smithi196.stdout: partition_filters: 0 2024-04-14T15:32:15.431 INFO:journalctl@ceph.mon.b.smithi196.stdout: use_delta_encoding: 1 2024-04-14T15:32:15.431 INFO:journalctl@ceph.mon.b.smithi196.stdout: filter_policy: bloomfilter 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout: whole_key_filtering: 1 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout: verify_compression: 0 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout: read_amp_bytes_per_bit: 0 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout: format_version: 5 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_index_compression: 1 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_align: 0 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout: max_auto_readahead_size: 262144 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout: prepopulate_block_cache: 0 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout: initial_auto_readahead_size: 8192 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_file_reads_for_auto_readahead: 2 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T15:32:15.432 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compression: NoCompression 2024-04-14T15:32:15.433 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T15:32:15.433 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T15:32:15.433 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T15:32:15.433 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.num_levels: 7 2024-04-14T15:32:15.433 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T15:32:15.433 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T15:32:15.433 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T15:32:15.433 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T15:32:15.433 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T15:32:15.434 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T15:32:15.434 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T15:32:15.434 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:32:15.434 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T15:32:15.434 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T15:32:15.434 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:32:15.434 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:32:15.434 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T15:32:15.434 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T15:32:15.434 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T15:32:15.434 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T15:32:15.435 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T15:32:15.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T15:32:15.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T15:32:15.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T15:32:15.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T15:32:15.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T15:32:15.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T15:32:15.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T15:32:15.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-14T15:32:15.436 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.arena_block_size: 1048576 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T15:32:15.437 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.inplace_update_support: 0 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.bloom_locality: 0 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.max_successive_merges: 0 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T15:32:15.438 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.ttl: 2592000 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.enable_blob_files: false 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.min_blob_size: 0 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.blob_file_starting_level: 0 2024-04-14T15:32:15.439 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14888, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 67978c58-7d42-4141-8e54-56126af8cb02 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108735017552, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108735022343, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 1077009, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14893, "largest_seqno": 15467, "table_properties": {"data_size": 1073604, "index_size": 1725, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 5934, "raw_average_key_size": 23, "raw_value_size": 1067992, "raw_average_value_size": 4271, "num_data_blocks": 79, "num_entries": 250, "num_filter_entries": 250, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713108735, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "67978c58-7d42-4141-8e54-56126af8cb02", "db_session_id": "4KVHZ1AZPC9NX6YEDV5L", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713108735022435, "job": 1, "event": "recovery_finished"} 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T15:32:15.440 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561973594000 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: DB pointer 0x56197357e000 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** DB Stats ** 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:32:15.441 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: L0 1/0 1.03 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 448.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: L6 1/0 11.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: Sum 2/0 12.45 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 448.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 448.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 448.9 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:32:15.442 INFO:journalctl@ceph.mon.b.smithi196.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative compaction: 0.00 GB write, 146.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval compaction: 0.00 GB write, 146.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache BinnedLRUCache@0x5619734a5090#2 capacity: 512.00 MB usage: 2.55 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.72 KB,0.000137091%) IndexBlock(1,1.83 KB,0.000348687%) Misc(1,0.00 KB,0%) 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T15:32:15.443 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: starting mon.b rank 2 at public addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] at bind addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:32:15.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: mon.b@-1(???) e3 preinit fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:32:15.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: mon.b@-1(???).mds e1 new map 2024-04-14T15:32:15.444 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: mon.b@-1(???).mds e1 print_map 2024-04-14T15:32:15.444 INFO:journalctl@ceph.mon.b.smithi196.stdout: e1 2024-04-14T15:32:15.444 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T15:32:15.444 INFO:journalctl@ceph.mon.b.smithi196.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T15:32:15.444 INFO:journalctl@ceph.mon.b.smithi196.stdout: legacy client fscid: -1 2024-04-14T15:32:15.444 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-04-14T15:32:15.444 INFO:journalctl@ceph.mon.b.smithi196.stdout: No filesystems configured 2024-04-14T15:32:15.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: mon.b@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-14T15:32:15.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T15:32:15.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T15:32:15.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T15:32:15.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 ceph-mon[58679]: mon.b@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-04-14T15:32:15.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 bash[58660]: 72817f80f666c7de49b830bec27e1ed08cf6ec896a527e08b538ec376a141bf4 2024-04-14T15:32:15.445 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:15 smithi196 systemd[1]: Started Ceph mon.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:18.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:20.361 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:32:20.030+0000 7fda5603a700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-14T15:32:20.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:20.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:32:20.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:32:20.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: fsmap 2024-04-14T15:32:20.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:32:20.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: mgrmap e40: y(active, since 33s), standbys: x 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: mon.a calling monitor election 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: mon.b calling monitor election 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: mon.c calling monitor election 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: mon.a is new leader, mons a,b in quorum (ranks 0,2) 2024-04-14T15:32:20.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: mon.a calling monitor election 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: monmap epoch 4 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: last_changed 2024-04-14T15:32:15.050009+0000 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: created 2024-04-14T15:18:56.351711+0000 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: min_mon_release 19 (squid) 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: election_strategy: 1 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: fsmap 2024-04-14T15:32:20.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: mgrmap e40: y(active, since 38s), standbys: x 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: fsmap 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: mgrmap e40: y(active, since 33s), standbys: x 2024-04-14T15:32:20.364 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: mon.a calling monitor election 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: mon.b calling monitor election 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: mon.c calling monitor election 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: mon.a is new leader, mons a,b in quorum (ranks 0,2) 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:20.365 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: mon.a calling monitor election 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: monmap epoch 4 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: last_changed 2024-04-14T15:32:15.050009+0000 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: created 2024-04-14T15:18:56.351711+0000 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: min_mon_release 19 (squid) 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: election_strategy: 1 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: fsmap 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: mgrmap e40: y(active, since 38s), standbys: x 2024-04-14T15:32:20.366 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:20.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:20.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:20.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:20.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:20.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:20.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: fsmap 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: mgrmap e40: y(active, since 33s), standbys: x 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: mon.a calling monitor election 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: mon.b calling monitor election 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: mon.c calling monitor election 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-04-14T15:32:20.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: mon.a is new leader, mons a,b in quorum (ranks 0,2) 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: mon.a calling monitor election 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: monmap epoch 4 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: last_changed 2024-04-14T15:32:15.050009+0000 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: created 2024-04-14T15:18:56.351711+0000 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: min_mon_release 19 (squid) 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: election_strategy: 1 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: 0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: 1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2024-04-14T15:32:20.417 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: 2: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-04-14T15:32:20.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: fsmap 2024-04-14T15:32:20.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T15:32:20.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: mgrmap e40: y(active, since 38s), standbys: x 2024-04-14T15:32:20.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:20.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:20.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:20.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:20.418 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:21.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:32:21 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:32:21] "GET /metrics HTTP/1.1" 200 38259 "" "Prometheus/2.33.4" 2024-04-14T15:32:21.981 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:21 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 9. 2024-04-14T15:32:21.981 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:21 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:21.981 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:21 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:32:22.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:22 smithi196 ceph-mon[58679]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:22.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:22 smithi016 ceph-mon[67348]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:22.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:22 smithi016 ceph-mon[64807]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:22.664 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:22 smithi196 podman[59066]: 2024-04-14 15:32:22.281793819 +0000 UTC m=+0.020164028 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:32:22.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:22 smithi196 podman[59066]: 2024-04-14 15:32:22.434558562 +0000 UTC m=+0.172928760 container create b71a5812138a90b923b6319434b6d159c4253fbb470c3134453d2247af445f72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, description=Ceph Grafana Container, distribution-scope=public, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, version=8.5, architecture=x86_64, vendor=Red Hat, Inc.) 2024-04-14T15:32:23.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:22 smithi196 podman[59066]: 2024-04-14 15:32:22.701977455 +0000 UTC m=+0.440347655 container init b71a5812138a90b923b6319434b6d159c4253fbb470c3134453d2247af445f72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, vendor=Red Hat, Inc., name=ubi8, description=Ceph Grafana Container, com.redhat.component=ubi8-container, version=8.5, build-date=2022-03-28T10:36:18.413762) 2024-04-14T15:32:23.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:22 smithi196 podman[59066]: 2024-04-14 15:32:22.705805537 +0000 UTC m=+0.444175737 container start b71a5812138a90b923b6319434b6d159c4253fbb470c3134453d2247af445f72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, distribution-scope=public, version=8.5, name=ubi8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, vcs-type=git, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762) 2024-04-14T15:32:23.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:22 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[59096]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:32:23.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:22 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[59096]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:32:23.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:22 smithi196 bash[59066]: b71a5812138a90b923b6319434b6d159c4253fbb470c3134453d2247af445f72 2024-04-14T15:32:23.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:22 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:23.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:22 smithi196 podman[59128]: 2024-04-14 15:32:22.816620691 +0000 UTC m=+0.073033574 container died b71a5812138a90b923b6319434b6d159c4253fbb470c3134453d2247af445f72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, release=236.1648460182, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , description=Ceph Grafana Container, vcs-type=git, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, io.openshift.tags=base rhel8) 2024-04-14T15:32:23.665 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:23.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:23 smithi196 podman[59128]: 2024-04-14 15:32:23.355867644 +0000 UTC m=+0.612280514 container remove b71a5812138a90b923b6319434b6d159c4253fbb470c3134453d2247af445f72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, name=ubi8, architecture=x86_64, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-04-14T15:32:23.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:23 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:32:23.665 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:23 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:32:24.387 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:24 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:24.388 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:24 smithi196 ceph-mon[58679]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:24.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:24 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:24.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:24 smithi016 ceph-mon[64807]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:24.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:24 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:24.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:24 smithi016 ceph-mon[67348]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:26.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:26 smithi196 ceph-mon[58679]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:26.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:26 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:26.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:26 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:26.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:26 smithi016 ceph-mon[67348]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:26.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:26 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:26.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:26 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:26.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:26 smithi016 ceph-mon[64807]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:26.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:26 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:26.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:26 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:27.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:27.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:27.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:27.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:27.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:27.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:27.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:28.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:28 smithi196 ceph-mon[58679]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:28.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:28 smithi016 ceph-mon[67348]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:28.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:28 smithi016 ceph-mon[64807]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:28.914 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:28 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:30.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:30.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:30.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:30.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:30.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:32:30.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-14T15:32:30.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:30.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:30.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: Reconfiguring mon.a (monmap changed)... 2024-04-14T15:32:30.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:30.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:30.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:30.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[64807]: Reconfiguring daemon mon.a on smithi016 2024-04-14T15:32:30.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:30.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:30.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:30.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:30.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:32:30.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-14T15:32:30.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:30.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:30.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: Reconfiguring mon.a (monmap changed)... 2024-04-14T15:32:30.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:30.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:30.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:30.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:30 smithi016 ceph-mon[67348]: Reconfiguring daemon mon.a on smithi016 2024-04-14T15:32:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:32:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-14T15:32:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:30.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:30.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: Reconfiguring mon.a (monmap changed)... 2024-04-14T15:32:30.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:30.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:30.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:30.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:30 smithi196 ceph-mon[58679]: Reconfiguring daemon mon.a on smithi016 2024-04-14T15:32:31.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:32:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:32:31] "GET /metrics HTTP/1.1" 200 38273 "" "Prometheus/2.33.4" 2024-04-14T15:32:32.837 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:32.837 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:32.837 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[67348]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-04-14T15:32:32.837 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[67348]: Reconfiguring mgr.y (monmap changed)... 2024-04-14T15:32:32.837 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:32:32.838 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:32:32.838 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:32.838 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[67348]: Reconfiguring daemon mgr.y on smithi016 2024-04-14T15:32:32.838 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:32.838 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:32.838 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[64807]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-04-14T15:32:32.838 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[64807]: Reconfiguring mgr.y (monmap changed)... 2024-04-14T15:32:32.838 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:32:32.838 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:32:32.838 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:32.838 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:32 smithi016 ceph-mon[64807]: Reconfiguring daemon mgr.y on smithi016 2024-04-14T15:32:32.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:32 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:32.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:32 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:32.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:32 smithi196 ceph-mon[58679]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-04-14T15:32:32.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:32 smithi196 ceph-mon[58679]: Reconfiguring mgr.y (monmap changed)... 2024-04-14T15:32:32.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:32 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:32:32.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:32 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:32:32.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:32 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:32.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:32 smithi196 ceph-mon[58679]: Reconfiguring daemon mgr.y on smithi016 2024-04-14T15:32:33.760 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:33.761 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 10. 2024-04-14T15:32:33.761 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:33.761 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:32:33.761 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 podman[59841]: 2024-04-14 15:32:33.553435076 +0000 UTC m=+0.017808264 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:32:33.761 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 podman[59841]: 2024-04-14 15:32:33.660726744 +0000 UTC m=+0.125099920 container create 3f7c92e49b04431c51b5320f16a9dc339ea6e476e3219851945efc97bb08f891 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-type=git, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, name=ubi8, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, architecture=x86_64, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-14T15:32:34.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 podman[59841]: 2024-04-14 15:32:33.880039513 +0000 UTC m=+0.344412691 container init 3f7c92e49b04431c51b5320f16a9dc339ea6e476e3219851945efc97bb08f891 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, architecture=x86_64, release=236.1648460182, io.buildah.version=1.24.2, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, name=ubi8, distribution-scope=public, io.openshift.expose-services=, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-14T15:32:34.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 podman[59841]: 2024-04-14 15:32:33.88266301 +0000 UTC m=+0.347036190 container start 3f7c92e49b04431c51b5320f16a9dc339ea6e476e3219851945efc97bb08f891 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, name=ubi8, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, vendor=Red Hat, Inc., release=236.1648460182, vcs-type=git, io.buildah.version=1.24.2, version=8.5, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-04-14T15:32:34.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[59856]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:32:34.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[59856]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:32:34.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 bash[59841]: 3f7c92e49b04431c51b5320f16a9dc339ea6e476e3219851945efc97bb08f891 2024-04-14T15:32:34.166 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:34.166 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:33 smithi196 podman[59872]: 2024-04-14 15:32:33.969026787 +0000 UTC m=+0.052274776 container died 3f7c92e49b04431c51b5320f16a9dc339ea6e476e3219851945efc97bb08f891 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, vendor=Red Hat, Inc., io.openshift.expose-services=, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, architecture=x86_64, io.openshift.tags=base rhel8, vcs-type=git, maintainer=Paul Cuzner , distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762) 2024-04-14T15:32:34.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[64807]: Reconfiguring mon.c (monmap changed)... 2024-04-14T15:32:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[64807]: Reconfiguring daemon mon.c on smithi016 2024-04-14T15:32:34.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[64807]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:34.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:34.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:34.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:34.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[67348]: Reconfiguring mon.c (monmap changed)... 2024-04-14T15:32:34.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:34.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:34.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:34.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[67348]: Reconfiguring daemon mon.c on smithi016 2024-04-14T15:32:34.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:34 smithi016 ceph-mon[67348]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:34.662 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:34 smithi196 podman[59872]: 2024-04-14 15:32:34.541832552 +0000 UTC m=+0.625080533 container remove 3f7c92e49b04431c51b5320f16a9dc339ea6e476e3219851945efc97bb08f891 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, version=8.5, vendor=Red Hat, Inc., architecture=x86_64, distribution-scope=public, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, description=Ceph Grafana Container, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=) 2024-04-14T15:32:34.663 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:34 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:32:34.663 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:34 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:32:34.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:34 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:34.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:34 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:34.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:34 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:34.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:34 smithi196 ceph-mon[58679]: Reconfiguring mon.c (monmap changed)... 2024-04-14T15:32:34.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:34 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:34.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:34 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:34.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:34 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:34.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:34 smithi196 ceph-mon[58679]: Reconfiguring daemon mon.c on smithi016 2024-04-14T15:32:34.663 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:34 smithi196 ceph-mon[58679]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:35 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:35 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:35 smithi196 ceph-mon[58679]: Reconfiguring osd.0 (monmap changed)... 2024-04-14T15:32:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:35 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:32:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:35 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:35 smithi196 ceph-mon[58679]: Reconfiguring daemon osd.0 on smithi016 2024-04-14T15:32:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:35 smithi196 ceph-mon[58679]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:36.213 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:32:36.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:36.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:36.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[64807]: Reconfiguring osd.0 (monmap changed)... 2024-04-14T15:32:36.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:32:36.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:36.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[64807]: Reconfiguring daemon osd.0 on smithi016 2024-04-14T15:32:36.214 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[64807]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:36.214 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:36.214 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:36.214 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[67348]: Reconfiguring osd.0 (monmap changed)... 2024-04-14T15:32:36.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:32:36.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:36.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[67348]: Reconfiguring daemon osd.0 on smithi016 2024-04-14T15:32:36.215 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:35 smithi016 ceph-mon[67348]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:36.934 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:32:36.934 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (2m) 32s ago 7m 17.7M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:32:36.934 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 unknown 11s ago 7m - - 2024-04-14T15:32:36.934 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (92s) 32s ago 6m 49.6M - 3.5 e1d6a67b021e 27d95fa4403f 2024-04-14T15:32:36.934 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (80s) 11s ago 11m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:32:36.934 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (117s) 32s ago 13m 494M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:32:36.934 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (65s) 32s ago 13m 39.1M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:32:36.934 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (22s) 11s ago 12m 22.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 72817f80f666 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (39s) 32s ago 12m 17.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (2m) 32s ago 8m 17.5M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (2m) 11s ago 7m 18.0M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (11m) 32s ago 11m 53.5M 2737M 17.2.0 e1d6a67b021e b812f3e0fa81 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (11m) 32s ago 11m 52.8M 2737M 17.2.0 e1d6a67b021e c275d1a0eaeb 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (10m) 32s ago 10m 51.0M 2737M 17.2.0 e1d6a67b021e 592ce301a74a 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (10m) 32s ago 10m 51.4M 2737M 17.2.0 e1d6a67b021e 1ed8730c3fc1 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (10m) 11s ago 10m 54.4M 2992M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (9m) 11s ago 9m 50.8M 2992M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (9m) 11s ago 9m 49.6M 2992M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (8m) 11s ago 8m 54.7M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:32:36.935 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (86s) 11s ago 8m 71.0M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:32:36.936 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (6m) 32s ago 6m 74.7M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:32:36.936 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (6m) 11s ago 6m 74.9M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:32:36.936 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (6m) 32s ago 6m 72.6M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:32:36.936 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (6m) 11s ago 6m 75.3M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:32:37.360 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:32:37.361 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-14T15:32:37.361 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 5 2024-04-14T15:32:37.361 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:32:37.361 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:32:37.713 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:32:37.713 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:32:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:32:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:32:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:32:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-04-14T15:32:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-14T15:32:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:32:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "5/23 daemons upgraded", 2024-04-14T15:32:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading mon daemons", 2024-04-14T15:32:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:32:37.714 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:32:37.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[64807]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[64807]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[64807]: Reconfiguring osd.1 (monmap changed)... 2024-04-14T15:32:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:32:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[64807]: Reconfiguring daemon osd.1 on smithi016 2024-04-14T15:32:37.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/2850634320' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:32:37.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[67348]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:37.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[67348]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:37.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:37.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:37.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[67348]: Reconfiguring osd.1 (monmap changed)... 2024-04-14T15:32:37.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:32:37.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:37.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[67348]: Reconfiguring daemon osd.1 on smithi016 2024-04-14T15:32:37.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:37 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/2850634320' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:32:37.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:37 smithi196 ceph-mon[58679]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:37 smithi196 ceph-mon[58679]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:37 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:37 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:37 smithi196 ceph-mon[58679]: Reconfiguring osd.1 (monmap changed)... 2024-04-14T15:32:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:37 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:32:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:37 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:37 smithi196 ceph-mon[58679]: Reconfiguring daemon osd.1 on smithi016 2024-04-14T15:32:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:37 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/2850634320' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:32:38.128 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:32:38.128 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:32:38.128 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:32:38.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:38.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:38 smithi196 ceph-mon[58679]: from='client.54151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:38.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:38 smithi196 ceph-mon[58679]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-04-14T15:32:38.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:38 smithi196 ceph-mon[58679]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:38.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:38 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/2518838679' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:32:38.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:38 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:38.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:38 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:38.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:38 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:32:38.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:38 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:38.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[64807]: from='client.54151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:38.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[64807]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-04-14T15:32:38.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[64807]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:38.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/2518838679' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:32:38.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:38.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:38.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:32:38.954 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:38.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[67348]: from='client.54151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:38.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[67348]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-04-14T15:32:38.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[67348]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:32:38.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/2518838679' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:32:38.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:38.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:38.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:32:38.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:38 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:39.901 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:39 smithi016 ceph-mon[64807]: Reconfiguring osd.2 (monmap changed)... 2024-04-14T15:32:39.901 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:39 smithi016 ceph-mon[64807]: Reconfiguring daemon osd.2 on smithi016 2024-04-14T15:32:39.901 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:39 smithi016 ceph-mon[67348]: Reconfiguring osd.2 (monmap changed)... 2024-04-14T15:32:39.901 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:39 smithi016 ceph-mon[67348]: Reconfiguring daemon osd.2 on smithi016 2024-04-14T15:32:39.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:39 smithi196 ceph-mon[58679]: Reconfiguring osd.2 (monmap changed)... 2024-04-14T15:32:39.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:39 smithi196 ceph-mon[58679]: Reconfiguring daemon osd.2 on smithi016 2024-04-14T15:32:40.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:40 smithi196 ceph-mon[58679]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:32:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:32:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:40.962 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:40 smithi016 ceph-mon[67348]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:32:40.962 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:40.962 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:40.962 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:32:40.962 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:40.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:40 smithi016 ceph-mon[64807]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:32:40.962 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:40.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:40.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:32:40.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:41.635 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:32:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:32:41] "GET /metrics HTTP/1.1" 200 38273 "" "Prometheus/2.33.4" 2024-04-14T15:32:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:41 smithi196 ceph-mon[58679]: Reconfiguring osd.3 (monmap changed)... 2024-04-14T15:32:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:41 smithi196 ceph-mon[58679]: Reconfiguring daemon osd.3 on smithi016 2024-04-14T15:32:42.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:41 smithi016 ceph-mon[67348]: Reconfiguring osd.3 (monmap changed)... 2024-04-14T15:32:42.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:41 smithi016 ceph-mon[67348]: Reconfiguring daemon osd.3 on smithi016 2024-04-14T15:32:42.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:41 smithi016 ceph-mon[64807]: Reconfiguring osd.3 (monmap changed)... 2024-04-14T15:32:42.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:41 smithi016 ceph-mon[64807]: Reconfiguring daemon osd.3 on smithi016 2024-04-14T15:32:42.686 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[67348]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:42.686 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:42.686 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:42.686 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:42.686 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:42.686 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:42.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[64807]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:42.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:42.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:42.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:42.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:42.687 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:42 smithi196 ceph-mon[58679]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:43.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[67348]: Reconfiguring rgw.foo.smithi016.qozpor (monmap changed)... 2024-04-14T15:32:43.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[67348]: Reconfiguring daemon rgw.foo.smithi016.qozpor on smithi016 2024-04-14T15:32:43.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:43.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:43.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:43.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:43.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[64807]: Reconfiguring rgw.foo.smithi016.qozpor (monmap changed)... 2024-04-14T15:32:43.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[64807]: Reconfiguring daemon rgw.foo.smithi016.qozpor on smithi016 2024-04-14T15:32:43.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:43.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:43.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:43.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:43 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:43 smithi196 ceph-mon[58679]: Reconfiguring rgw.foo.smithi016.qozpor (monmap changed)... 2024-04-14T15:32:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:43 smithi196 ceph-mon[58679]: Reconfiguring daemon rgw.foo.smithi016.qozpor on smithi016 2024-04-14T15:32:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:43 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:43 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:43 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:43 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:43.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:44.855 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:44 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:44.856 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:44 smithi016 ceph-mon[67348]: Reconfiguring rgw.smpl.smithi016.sjsxjs (monmap changed)... 2024-04-14T15:32:44.856 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:44 smithi016 ceph-mon[67348]: Reconfiguring daemon rgw.smpl.smithi016.sjsxjs on smithi016 2024-04-14T15:32:44.856 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:44 smithi016 ceph-mon[67348]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:44.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:44 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:44.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:44 smithi016 ceph-mon[64807]: Reconfiguring rgw.smpl.smithi016.sjsxjs (monmap changed)... 2024-04-14T15:32:44.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:44 smithi016 ceph-mon[64807]: Reconfiguring daemon rgw.smpl.smithi016.sjsxjs on smithi016 2024-04-14T15:32:44.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:44 smithi016 ceph-mon[64807]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:44.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:44 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:44.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:44 smithi196 ceph-mon[58679]: Reconfiguring rgw.smpl.smithi016.sjsxjs (monmap changed)... 2024-04-14T15:32:44.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:44 smithi196 ceph-mon[58679]: Reconfiguring daemon rgw.smpl.smithi016.sjsxjs on smithi016 2024-04-14T15:32:44.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:44 smithi196 ceph-mon[58679]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:44.896 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:44 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 11. 2024-04-14T15:32:44.896 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:44 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:44.896 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:44 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:32:45.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:44 smithi196 podman[59942]: 2024-04-14 15:32:44.798488718 +0000 UTC m=+0.017560845 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:32:45.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:44 smithi196 podman[59942]: 2024-04-14 15:32:44.950623048 +0000 UTC m=+0.169695159 container create 38f42bd5244e67f4154ddec34640d9c55afcb192c78dc6cfc96dc6b838301922 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, distribution-scope=public, maintainer=Paul Cuzner , version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, vcs-type=git, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, release=236.1648460182, name=ubi8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-14T15:32:45.525 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:45 smithi196 podman[59942]: 2024-04-14 15:32:45.205203013 +0000 UTC m=+0.424275120 container init 38f42bd5244e67f4154ddec34640d9c55afcb192c78dc6cfc96dc6b838301922 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, vcs-type=git, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, name=ubi8, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, release=236.1648460182, version=8.5, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-14T15:32:45.525 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:45 smithi196 podman[59942]: 2024-04-14 15:32:45.207862491 +0000 UTC m=+0.426934604 container start 38f42bd5244e67f4154ddec34640d9c55afcb192c78dc6cfc96dc6b838301922 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.openshift.tags=base rhel8, version=8.5, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, maintainer=Paul Cuzner , release=236.1648460182, name=ubi8, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-14T15:32:45.526 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[59957]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:32:45.526 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[59957]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:32:45.526 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:45 smithi196 bash[59942]: 38f42bd5244e67f4154ddec34640d9c55afcb192c78dc6cfc96dc6b838301922 2024-04-14T15:32:45.526 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:45 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:45.526 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:45 smithi196 podman[59973]: 2024-04-14 15:32:45.316446695 +0000 UTC m=+0.075710232 container died 38f42bd5244e67f4154ddec34640d9c55afcb192c78dc6cfc96dc6b838301922 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vcs-type=git, release=236.1648460182, architecture=x86_64, vendor=Red Hat, Inc., name=ubi8, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-04-14T15:32:46.164 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:45 smithi196 podman[59973]: 2024-04-14 15:32:45.892232258 +0000 UTC m=+0.651495789 container remove 38f42bd5244e67f4154ddec34640d9c55afcb192c78dc6cfc96dc6b838301922 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, name=ubi8, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, distribution-scope=public, vcs-type=git, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182) 2024-04-14T15:32:46.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:45 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:32:46.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:45 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:32:46.458 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:46 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:46.458 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:46 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:46.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:46 smithi196 ceph-mon[58679]: Reconfiguring mon.b (monmap changed)... 2024-04-14T15:32:46.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:46 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:46.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:46 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:46.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:46 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:46.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:46 smithi196 ceph-mon[58679]: Reconfiguring daemon mon.b on smithi196 2024-04-14T15:32:46.459 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:46 smithi196 ceph-mon[58679]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:46.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:46.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:46.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[64807]: Reconfiguring mon.b (monmap changed)... 2024-04-14T15:32:46.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:46.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:46.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:46.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[64807]: Reconfiguring daemon mon.b on smithi196 2024-04-14T15:32:46.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[64807]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:46.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:46.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:46.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[67348]: Reconfiguring mon.b (monmap changed)... 2024-04-14T15:32:46.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T15:32:46.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T15:32:46.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:46.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[67348]: Reconfiguring daemon mon.b on smithi196 2024-04-14T15:32:46.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:46 smithi016 ceph-mon[67348]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:48.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:48.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:48.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:48 smithi196 ceph-mon[58679]: Reconfiguring mgr.x (monmap changed)... 2024-04-14T15:32:48.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:32:48.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:32:48.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:48.339 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:48 smithi196 ceph-mon[58679]: Reconfiguring daemon mgr.x on smithi196 2024-04-14T15:32:48.340 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:48 smithi196 ceph-mon[58679]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:48.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:48.360 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:48.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[64807]: Reconfiguring mgr.x (monmap changed)... 2024-04-14T15:32:48.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:32:48.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:32:48.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:48.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[64807]: Reconfiguring daemon mgr.x on smithi196 2024-04-14T15:32:48.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[64807]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:48.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:48.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:48.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[67348]: Reconfiguring mgr.x (monmap changed)... 2024-04-14T15:32:48.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T15:32:48.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T15:32:48.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:48.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[67348]: Reconfiguring daemon mgr.x on smithi196 2024-04-14T15:32:48.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:48 smithi016 ceph-mon[67348]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:48.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:49.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:49 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:49.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:49 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:49.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:49 smithi196 ceph-mon[58679]: Reconfiguring osd.4 (monmap changed)... 2024-04-14T15:32:49.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:49 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:32:49.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:49 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:49.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:49 smithi196 ceph-mon[58679]: Reconfiguring daemon osd.4 on smithi196 2024-04-14T15:32:50.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[67348]: Reconfiguring osd.4 (monmap changed)... 2024-04-14T15:32:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:32:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[67348]: Reconfiguring daemon osd.4 on smithi196 2024-04-14T15:32:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[64807]: Reconfiguring osd.4 (monmap changed)... 2024-04-14T15:32:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:32:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:50.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:49 smithi016 ceph-mon[64807]: Reconfiguring daemon osd.4 on smithi196 2024-04-14T15:32:50.946 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:50 smithi196 ceph-mon[58679]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:50.946 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:50 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:50.946 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:50 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:50.946 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:50 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:32:50.947 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:50 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:51.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:50 smithi016 ceph-mon[64807]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:50 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:50 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:50 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:32:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:50 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:51.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:50 smithi016 ceph-mon[67348]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:51.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:50 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:51.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:50 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:51.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:50 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:32:51.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:50 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:51.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:51 smithi016 ceph-mon[67348]: Reconfiguring osd.5 (monmap changed)... 2024-04-14T15:32:51.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:51 smithi016 ceph-mon[67348]: Reconfiguring daemon osd.5 on smithi196 2024-04-14T15:32:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:51 smithi016 ceph-mon[64807]: Reconfiguring osd.5 (monmap changed)... 2024-04-14T15:32:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:51 smithi016 ceph-mon[64807]: Reconfiguring daemon osd.5 on smithi196 2024-04-14T15:32:51.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:32:51 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:32:51] "GET /metrics HTTP/1.1" 200 38202 "" "Prometheus/2.33.4" 2024-04-14T15:32:51.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:51 smithi196 ceph-mon[58679]: Reconfiguring osd.5 (monmap changed)... 2024-04-14T15:32:51.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:51 smithi196 ceph-mon[58679]: Reconfiguring daemon osd.5 on smithi196 2024-04-14T15:32:52.981 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:52 smithi196 ceph-mon[58679]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:52.981 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:52 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:52.981 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:52 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:52.981 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:52 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:32:52.981 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:52 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:53.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:52 smithi016 ceph-mon[67348]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:53.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:52 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:53.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:52 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:53.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:52 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:32:53.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:52 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:52 smithi016 ceph-mon[64807]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:52 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:52 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:52 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:32:53.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:52 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:53.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:53 smithi196 ceph-mon[58679]: Reconfiguring osd.6 (monmap changed)... 2024-04-14T15:32:53.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:53 smithi196 ceph-mon[58679]: Reconfiguring daemon osd.6 on smithi196 2024-04-14T15:32:53.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:53 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:54.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:53 smithi016 ceph-mon[67348]: Reconfiguring osd.6 (monmap changed)... 2024-04-14T15:32:54.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:53 smithi016 ceph-mon[67348]: Reconfiguring daemon osd.6 on smithi196 2024-04-14T15:32:54.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:53 smithi016 ceph-mon[64807]: Reconfiguring osd.6 (monmap changed)... 2024-04-14T15:32:54.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:53 smithi016 ceph-mon[64807]: Reconfiguring daemon osd.6 on smithi196 2024-04-14T15:32:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:54 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:54 smithi196 ceph-mon[58679]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:54 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:54 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:54 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:32:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:54 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:55.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[64807]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:32:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:55.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:32:55.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[67348]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:32:55.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:55.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:55.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:32:55.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:54 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:56.011 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:55 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 12. 2024-04-14T15:32:56.012 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:55 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:56.012 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:55 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:32:56.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:55 smithi196 ceph-mon[58679]: Reconfiguring osd.7 (monmap changed)... 2024-04-14T15:32:56.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:55 smithi196 ceph-mon[58679]: Reconfiguring daemon osd.7 on smithi196 2024-04-14T15:32:56.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:55 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:56.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:55 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:56.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:55 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:56.013 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:55 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[64807]: Reconfiguring osd.7 (monmap changed)... 2024-04-14T15:32:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[64807]: Reconfiguring daemon osd.7 on smithi196 2024-04-14T15:32:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:56.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[67348]: Reconfiguring osd.7 (monmap changed)... 2024-04-14T15:32:56.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[67348]: Reconfiguring daemon osd.7 on smithi196 2024-04-14T15:32:56.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:56.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:56.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:56.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:55 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:56 smithi196 ceph-mon[58679]: Reconfiguring rgw.foo.smithi196.wjbrhf (monmap changed)... 2024-04-14T15:32:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:56 smithi196 ceph-mon[58679]: Reconfiguring daemon rgw.foo.smithi196.wjbrhf on smithi196 2024-04-14T15:32:57.040 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:56 smithi196 ceph-mon[58679]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:57.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:56 smithi016 ceph-mon[64807]: Reconfiguring rgw.foo.smithi196.wjbrhf (monmap changed)... 2024-04-14T15:32:57.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:56 smithi016 ceph-mon[64807]: Reconfiguring daemon rgw.foo.smithi196.wjbrhf on smithi196 2024-04-14T15:32:57.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:56 smithi016 ceph-mon[64807]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:57.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:56 smithi016 ceph-mon[67348]: Reconfiguring rgw.foo.smithi196.wjbrhf (monmap changed)... 2024-04-14T15:32:57.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:56 smithi016 ceph-mon[67348]: Reconfiguring daemon rgw.foo.smithi196.wjbrhf on smithi196 2024-04-14T15:32:57.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:56 smithi016 ceph-mon[67348]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:57.407 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:57 smithi196 podman[60994]: 2024-04-14 15:32:57.283181513 +0000 UTC m=+0.112463013 container create 51a0b4878f313b7d7f1ad32585f353775d8bf8daa0f2411a273a8521c372976d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.buildah.version=1.24.2, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vcs-type=git, version=8.5, name=ubi8, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762) 2024-04-14T15:32:57.408 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:57 smithi196 podman[60994]: 2024-04-14 15:32:57.188336882 +0000 UTC m=+0.017618382 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:32:57.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:57.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:57.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:57.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:57.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:57.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:57 smithi196 podman[60994]: 2024-04-14 15:32:57.569315409 +0000 UTC m=+0.398596909 container init 51a0b4878f313b7d7f1ad32585f353775d8bf8daa0f2411a273a8521c372976d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, com.redhat.component=ubi8-container, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, maintainer=Paul Cuzner , io.buildah.version=1.24.2, description=Ceph Grafana Container, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., version=8.5, architecture=x86_64) 2024-04-14T15:32:57.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:57 smithi196 podman[60994]: 2024-04-14 15:32:57.572510458 +0000 UTC m=+0.401791959 container start 51a0b4878f313b7d7f1ad32585f353775d8bf8daa0f2411a273a8521c372976d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, architecture=x86_64, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, distribution-scope=public, io.buildah.version=1.24.2, io.openshift.expose-services=) 2024-04-14T15:32:57.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[61027]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:32:57.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[61027]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:32:57.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:57 smithi196 bash[60994]: 51a0b4878f313b7d7f1ad32585f353775d8bf8daa0f2411a273a8521c372976d 2024-04-14T15:32:57.916 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:57 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:32:57.916 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:57 smithi196 podman[61055]: 2024-04-14 15:32:57.680716043 +0000 UTC m=+0.072490277 container died 51a0b4878f313b7d7f1ad32585f353775d8bf8daa0f2411a273a8521c372976d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.openshift.expose-services=, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.openshift.tags=base rhel8, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.component=ubi8-container, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.buildah.version=1.24.2, vendor=Red Hat, Inc., distribution-scope=public, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container) 2024-04-14T15:32:58.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:58.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:58.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:58.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:58.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:32:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:32:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:32:58.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:32:58.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:58 smithi196 podman[61055]: 2024-04-14 15:32:58.197758808 +0000 UTC m=+0.589533029 container remove 51a0b4878f313b7d7f1ad32585f353775d8bf8daa0f2411a273a8521c372976d (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, vendor=Red Hat, Inc., maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.component=ubi8-container, release=236.1648460182, version=8.5, architecture=x86_64, vcs-type=git, build-date=2022-03-28T10:36:18.413762, name=ubi8) 2024-04-14T15:32:58.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:58 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:32:58.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:32:58 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:32:58.727 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:32:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:32:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:32:59.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:58 smithi196 ceph-mon[58679]: Reconfiguring rgw.smpl.smithi196.uwpszz (monmap changed)... 2024-04-14T15:32:59.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:58 smithi196 ceph-mon[58679]: Reconfiguring daemon rgw.smpl.smithi196.uwpszz on smithi196 2024-04-14T15:32:59.104 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:32:58 smithi196 ceph-mon[58679]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:59.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:58 smithi016 ceph-mon[64807]: Reconfiguring rgw.smpl.smithi196.uwpszz (monmap changed)... 2024-04-14T15:32:59.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:58 smithi016 ceph-mon[64807]: Reconfiguring daemon rgw.smpl.smithi196.uwpszz on smithi196 2024-04-14T15:32:59.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:32:58 smithi016 ceph-mon[64807]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:32:59.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:58 smithi016 ceph-mon[67348]: Reconfiguring rgw.smpl.smithi196.uwpszz (monmap changed)... 2024-04-14T15:32:59.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:58 smithi016 ceph-mon[67348]: Reconfiguring daemon rgw.smpl.smithi196.uwpszz on smithi196 2024-04-14T15:32:59.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:32:58 smithi016 ceph-mon[67348]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:33:00.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:33:00.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:00.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:00.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: Upgrade: Setting container_image for all mon 2024-04-14T15:33:00.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T15:33:00.729 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: Upgrade: Setting container_image for all crash 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: Upgrade: osd.0 is safe to restart 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[64807]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:00.730 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: Upgrade: Setting container_image for all mon 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: Upgrade: Setting container_image for all crash 2024-04-14T15:33:00.731 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.732 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T15:33:00.732 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T15:33:00.732 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: Upgrade: osd.0 is safe to restart 2024-04-14T15:33:00.732 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:00 smithi016 ceph-mon[67348]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:33:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: Upgrade: Setting container_image for all mon 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: Upgrade: Setting container_image for all crash 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: Upgrade: osd.0 is safe to restart 2024-04-14T15:33:00.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:00 smithi196 ceph-mon[58679]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:01.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:33:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:33:01] "GET /metrics HTTP/1.1" 200 38269 "" "Prometheus/2.33.4" 2024-04-14T15:33:02.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[67348]: Upgrade: Updating osd.0 2024-04-14T15:33:02.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:02.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:33:02.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:02.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[67348]: Deploying daemon osd.0 on smithi016 2024-04-14T15:33:02.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[67348]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:33:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[64807]: Upgrade: Updating osd.0 2024-04-14T15:33:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:33:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[64807]: Deploying daemon osd.0 on smithi016 2024-04-14T15:33:02.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:02 smithi016 ceph-mon[64807]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:33:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:02 smithi196 ceph-mon[58679]: Upgrade: Updating osd.0 2024-04-14T15:33:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:02 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:02 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T15:33:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:02 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:02 smithi196 ceph-mon[58679]: Deploying daemon osd.0 on smithi016 2024-04-14T15:33:02.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:02 smithi196 ceph-mon[58679]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:33:03.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:33:04.611 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:04 smithi016 systemd[1]: Stopping Ceph osd.0 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:33:04.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:04 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:04 smithi196 ceph-mon[58679]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:04 smithi196 ceph-mon[58679]: osd.0 marked itself down and dead 2024-04-14T15:33:05.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:04 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:05.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:04 smithi016 ceph-mon[67348]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:05.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:04 smithi016 ceph-mon[67348]: osd.0 marked itself down and dead 2024-04-14T15:33:05.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:04 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:05.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:04 smithi016 ceph-mon[64807]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:05.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:04 smithi016 ceph-mon[64807]: osd.0 marked itself down and dead 2024-04-14T15:33:05.112 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[32383]: 2024-04-14T15:33:04.633+0000 7f1f0d50a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:33:05.112 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[32383]: 2024-04-14T15:33:04.633+0000 7f1f0d50a700 -1 osd.0 105 *** Got signal Terminated *** 2024-04-14T15:33:05.112 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[32383]: 2024-04-14T15:33:04.633+0000 7f1f0d50a700 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:33:05.459 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:05 smithi016 podman[70260]: 2024-04-14 15:33:05.175553153 +0000 UTC m=+0.652682764 container died b812f3e0fa81b5f49f445b0601c0e645290df777505b05e97a1dda1141e4139d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , distribution-scope=public, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754) 2024-04-14T15:33:05.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:05 smithi016 ceph-mon[67348]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:33:05.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:05 smithi016 ceph-mon[67348]: osdmap e106: 8 total, 7 up, 8 in 2024-04-14T15:33:05.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:05 smithi016 ceph-mon[64807]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:33:05.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:05 smithi016 ceph-mon[64807]: osdmap e106: 8 total, 7 up, 8 in 2024-04-14T15:33:05.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:05 smithi016 podman[70260]: 2024-04-14 15:33:05.633869852 +0000 UTC m=+1.110999470 container cleanup b812f3e0fa81b5f49f445b0601c0e645290df777505b05e97a1dda1141e4139d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, architecture=x86_64, ceph=True, release=754, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, RELEASE=HEAD, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-14T15:33:05.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:05 smithi016 bash[70260]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0 2024-04-14T15:33:05.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:05 smithi196 ceph-mon[58679]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:33:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:05 smithi196 ceph-mon[58679]: osdmap e106: 8 total, 7 up, 8 in 2024-04-14T15:33:06.314 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:05 smithi016 podman[70277]: 2024-04-14 15:33:05.967872298 +0000 UTC m=+0.794614727 container remove b812f3e0fa81b5f49f445b0601c0e645290df777505b05e97a1dda1141e4139d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0, vcs-type=git, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, name=centos-stream, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.openshift.expose-services=, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public) 2024-04-14T15:33:06.598 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:06 smithi016 podman[70370]: 2024-04-14 15:33:06.217177662 +0000 UTC m=+0.018052335 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:33:06.598 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:06 smithi016 podman[70370]: 2024-04-14 15:33:06.320076698 +0000 UTC m=+0.120951372 container create 49c517cb76c534aaaa7a74871287929f18422704b537c1df4cd9f8bed312e8c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, ceph=True, CEPH_POINT_RELEASE=) 2024-04-14T15:33:06.598 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:06 smithi016 podman[70370]: 2024-04-14 15:33:06.523106379 +0000 UTC m=+0.323981049 container init 49c517cb76c534aaaa7a74871287929f18422704b537c1df4cd9f8bed312e8c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-14T15:33:06.598 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:06 smithi016 podman[70370]: 2024-04-14 15:33:06.526190616 +0000 UTC m=+0.327065293 container start 49c517cb76c534aaaa7a74871287929f18422704b537c1df4cd9f8bed312e8c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5) 2024-04-14T15:33:06.598 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:06 smithi016 podman[70370]: 2024-04-14 15:33:06.597974136 +0000 UTC m=+0.398848810 container attach 49c517cb76c534aaaa7a74871287929f18422704b537c1df4cd9f8bed312e8c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-04-14T15:33:06.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:06 smithi016 ceph-mon[64807]: pgmap v50: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:33:06.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:06 smithi016 ceph-mon[64807]: osdmap e107: 8 total, 7 up, 8 in 2024-04-14T15:33:06.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:06 smithi016 ceph-mon[67348]: pgmap v50: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:33:06.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:06 smithi016 ceph-mon[67348]: osdmap e107: 8 total, 7 up, 8 in 2024-04-14T15:33:06.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:06 smithi016 podman[70370]: 2024-04-14 15:33:06.833148665 +0000 UTC m=+0.634023336 container died 49c517cb76c534aaaa7a74871287929f18422704b537c1df4cd9f8bed312e8c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-14T15:33:06.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:06 smithi196 ceph-mon[58679]: pgmap v50: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:33:06.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:06 smithi196 ceph-mon[58679]: osdmap e107: 8 total, 7 up, 8 in 2024-04-14T15:33:07.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:07 smithi016 podman[70396]: 2024-04-14 15:33:07.586347934 +0000 UTC m=+0.748116464 container remove 49c517cb76c534aaaa7a74871287929f18422704b537c1df4cd9f8bed312e8c6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-14T15:33:07.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:07 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-14T15:33:07.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:07 smithi016 systemd[1]: Stopped Ceph osd.0 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:33:07.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:07 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.0.service: Consumed 7.290s CPU time. 2024-04-14T15:33:07.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:07 smithi016 systemd[1]: Starting Ceph osd.0 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:33:08.282 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 podman[70501]: 2024-04-14 15:33:07.933070467 +0000 UTC m=+0.018125924 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:33:08.282 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 podman[70501]: 2024-04-14 15:33:08.054274397 +0000 UTC m=+0.139329834 container create 9adde2d6691945e040dd8c01cbf7b02e709cbda934e608091e8f8b08b2df7276 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.build-date=20240403) 2024-04-14T15:33:08.283 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 podman[70501]: 2024-04-14 15:33:08.282098433 +0000 UTC m=+0.367153886 container init 9adde2d6691945e040dd8c01cbf7b02e709cbda934e608091e8f8b08b2df7276 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-a971fd5) 2024-04-14T15:33:08.500 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:33:08.610 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 podman[70501]: 2024-04-14 15:33:08.287456185 +0000 UTC m=+0.372511632 container start 9adde2d6691945e040dd8c01cbf7b02e709cbda934e608091e8f8b08b2df7276 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:33:08.611 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 podman[70501]: 2024-04-14 15:33:08.373784523 +0000 UTC m=+0.458839980 container attach 9adde2d6691945e040dd8c01cbf7b02e709cbda934e608091e8f8b08b2df7276 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate, GIT_CLEAN=True, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-14T15:33:08.671 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:08 smithi196 ceph-mon[58679]: pgmap v52: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:33:08.672 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:33:08.672 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:08 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 13. 2024-04-14T15:33:08.672 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:08 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:33:08.672 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:08 smithi196 systemd[1]: Starting Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:33:08.672 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:08 smithi196 podman[61184]: 2024-04-14 15:33:08.553177197 +0000 UTC m=+0.017296285 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T15:33:08.949 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:08 smithi196 podman[61184]: 2024-04-14 15:33:08.671827578 +0000 UTC m=+0.135946682 container create 0236eb6f192a057894bf52a90e8dee768b6feb418046e383dc704d590daf21a4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, name=ubi8, vcs-type=git, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, io.openshift.expose-services=, architecture=x86_64) 2024-04-14T15:33:09.060 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:08 smithi016 ceph-mon[67348]: pgmap v52: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:33:09.061 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:08 smithi016 ceph-mon[64807]: pgmap v52: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:33:09.061 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate[70518]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T15:33:09.061 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate[70518]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T15:33:09.061 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 bash[70501]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T15:33:09.061 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 bash[70501]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T15:33:09.061 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 bash[70501]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T15:33:09.061 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 bash[70501]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T15:33:09.061 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 bash[70501]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-14T15:33:09.061 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 bash[70501]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T15:33:09.062 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 bash[70501]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-14T15:33:09.062 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate[70518]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T15:33:09.062 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate[70518]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T15:33:09.062 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate[70518]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-14T15:33:09.062 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate[70518]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T15:33:09.062 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate[70518]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-14T15:33:09.062 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:08 smithi016 podman[70501]: 2024-04-14 15:33:08.765870108 +0000 UTC m=+0.850925548 container died 9adde2d6691945e040dd8c01cbf7b02e709cbda934e608091e8f8b08b2df7276 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:33:09.187 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:33:09.187 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (3m) 64s ago 8m 17.7M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:33:09.187 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 unknown 44s ago 7m - - 2024-04-14T15:33:09.187 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (2m) 64s ago 6m 49.6M - 3.5 e1d6a67b021e 27d95fa4403f 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (112s) 44s ago 12m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (2m) 64s ago 14m 494M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (97s) 64s ago 14m 39.1M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (54s) 44s ago 12m 22.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 72817f80f666 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (71s) 64s ago 12m 17.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (3m) 64s ago 8m 17.5M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (2m) 44s ago 8m 18.0M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (12m) 64s ago 12m 53.5M 2737M 17.2.0 e1d6a67b021e b812f3e0fa81 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (11m) 64s ago 11m 52.8M 2737M 17.2.0 e1d6a67b021e c275d1a0eaeb 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (11m) 64s ago 11m 51.0M 2737M 17.2.0 e1d6a67b021e 592ce301a74a 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (10m) 64s ago 10m 51.4M 2737M 17.2.0 e1d6a67b021e 1ed8730c3fc1 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (10m) 44s ago 10m 54.4M 2992M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (10m) 44s ago 10m 50.8M 2992M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:33:09.188 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (9m) 44s ago 9m 49.6M 2992M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:33:09.189 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (9m) 44s ago 9m 54.7M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:33:09.189 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (118s) 44s ago 8m 71.0M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:33:09.189 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (7m) 64s ago 7m 74.7M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:33:09.189 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (7m) 44s ago 7m 74.9M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:33:09.189 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (7m) 64s ago 7m 72.6M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:33:09.189 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (7m) 44s ago 7m 75.3M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:33:09.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:08 smithi196 podman[61184]: 2024-04-14 15:33:08.949178083 +0000 UTC m=+0.413297145 container init 0236eb6f192a057894bf52a90e8dee768b6feb418046e383dc704d590daf21a4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, distribution-scope=public, com.redhat.component=ubi8-container, vcs-type=git, io.openshift.expose-services=, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762) 2024-04-14T15:33:09.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:08 smithi196 podman[61184]: 2024-04-14 15:33:08.952113661 +0000 UTC m=+0.416232732 container start 0236eb6f192a057894bf52a90e8dee768b6feb418046e383dc704d590daf21a4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, distribution-scope=public, version=8.5, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, release=236.1648460182, io.openshift.tags=base rhel8, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, io.openshift.expose-services=, maintainer=Paul Cuzner ) 2024-04-14T15:33:09.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[61199]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:33:09.415 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a[61199]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T15:33:09.416 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:09 smithi196 bash[61184]: 0236eb6f192a057894bf52a90e8dee768b6feb418046e383dc704d590daf21a4 2024-04-14T15:33:09.416 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:09 smithi196 systemd[1]: Started Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:33:09.416 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:09 smithi196 podman[61215]: 2024-04-14 15:33:09.057832623 +0000 UTC m=+0.073003990 container died 0236eb6f192a057894bf52a90e8dee768b6feb418046e383dc704d590daf21a4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, distribution-scope=public, architecture=x86_64, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-14T15:33:09.610 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:33:09.610 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:33:09.610 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:33:09.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 5 2024-04-14T15:33:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:33:09.612 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:33:09.851 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:09 smithi016 ceph-mon[67348]: from='client.44208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:09.851 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:09 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/3197846919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:09.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:09 smithi016 ceph-mon[64807]: from='client.44208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:09.852 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:09 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/3197846919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:09.852 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:09 smithi016 podman[70686]: 2024-04-14 15:33:09.495264128 +0000 UTC m=+0.723946278 container remove 9adde2d6691945e040dd8c01cbf7b02e709cbda934e608091e8f8b08b2df7276 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:33:09.852 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:09 smithi016 podman[70796]: 2024-04-14 15:33:09.61227317 +0000 UTC m=+0.018309367 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:33:09.852 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:09 smithi016 podman[70796]: 2024-04-14 15:33:09.744955116 +0000 UTC m=+0.150991312 container create f6ee7262903f6081e4ceb07846511851ebf40d52a7dbc809ad9aa00d95e31b57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True) 2024-04-14T15:33:09.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:09 smithi196 ceph-mon[58679]: from='client.44208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:09.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:09 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/3197846919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:09.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:09 smithi196 podman[61215]: 2024-04-14 15:33:09.619545819 +0000 UTC m=+0.634717184 container remove 0236eb6f192a057894bf52a90e8dee768b6feb418046e383dc704d590daf21a4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, io.buildah.version=1.24.2, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , vcs-type=git, release=236.1648460182, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, io.openshift.tags=base rhel8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container) 2024-04-14T15:33:09.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:09 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T15:33:09.915 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:09 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:33:09.966 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:33:09.966 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:33:09.966 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:33:09.967 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:33:09.967 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:33:09.967 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-04-14T15:33:09.967 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-14T15:33:09.967 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:33:09.967 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "5/23 daemons upgraded", 2024-04-14T15:33:09.967 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T15:33:09.967 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:33:09.967 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:33:10.110 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:09 smithi016 podman[70796]: 2024-04-14 15:33:09.974670661 +0000 UTC m=+0.380706857 container init f6ee7262903f6081e4ceb07846511851ebf40d52a7dbc809ad9aa00d95e31b57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-14T15:33:10.111 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:09 smithi016 podman[70796]: 2024-04-14 15:33:09.977789347 +0000 UTC m=+0.383825550 container start f6ee7262903f6081e4ceb07846511851ebf40d52a7dbc809ad9aa00d95e31b57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240403, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-14T15:33:10.111 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:09 smithi016 ceph-osd[70841]: -- 172.21.15.16:0/3138735621 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5645fc8ed680 con 0x5645fc8b7000 2024-04-14T15:33:10.111 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:10 smithi016 bash[70796]: f6ee7262903f6081e4ceb07846511851ebf40d52a7dbc809ad9aa00d95e31b57 2024-04-14T15:33:10.111 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:10 smithi016 systemd[1]: Started Ceph osd.0 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:33:10.395 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 68/627 objects degraded (10.845%), 20 pgs degraded 2024-04-14T15:33:10.395 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:33:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:33:10.395 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-14T15:33:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: osd.0 (root=default,host=smithi016) is down 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 68/627 objects degraded (10.845%), 20 pgs degraded 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.8 is active+undersized+degraded, acting [7,1] 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.c is active+undersized+degraded, acting [2,5] 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.f is active+undersized+degraded, acting [4,7] 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.1e is active+undersized+degraded, acting [3,5] 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-04-14T15:33:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-04-14T15:33:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-04-14T15:33:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-04-14T15:33:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-04-14T15:33:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-04-14T15:33:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-04-14T15:33:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-04-14T15:33:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-04-14T15:33:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-04-14T15:33:10.666 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[70835]: 2024-04-14T15:33:10.516+0000 7f891790b7c0 -1 Falling back to public interface 2024-04-14T15:33:10.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[64807]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:10.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[64807]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:10.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[64807]: pgmap v53: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:10.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[64807]: Health check failed: Degraded data redundancy: 68/627 objects degraded (10.845%), 20 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:10.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:10 smithi196 ceph-mon[58679]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:10.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:10 smithi196 ceph-mon[58679]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:10.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:10 smithi196 ceph-mon[58679]: pgmap v53: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:10.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:10 smithi196 ceph-mon[58679]: Health check failed: Degraded data redundancy: 68/627 objects degraded (10.845%), 20 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:10.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:10 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:10.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:10 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:10.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:10 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/3035597627' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:33:10.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:10.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:10.945 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/3035597627' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:33:10.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[67348]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:10.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[67348]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:10.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[67348]: pgmap v53: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:10.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[67348]: Health check failed: Degraded data redundancy: 68/627 objects degraded (10.845%), 20 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:10.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:10.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:10.946 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:10 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/3035597627' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:33:11.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:11 smithi016 ceph-mon[67348]: from='client.54199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:11.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:11 smithi016 ceph-mon[64807]: from='client.54199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:11.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:33:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:33:11] "GET /metrics HTTP/1.1" 200 38269 "" "Prometheus/2.33.4" 2024-04-14T15:33:12.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:11 smithi196 ceph-mon[58679]: from='client.54199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:12.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:12 smithi016 ceph-mon[64807]: pgmap v54: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-04-14T15:33:12.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:12.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:12.911 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:12 smithi016 ceph-mon[67348]: pgmap v54: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-04-14T15:33:12.911 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:12.911 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:13.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:12 smithi196 ceph-mon[58679]: pgmap v54: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-04-14T15:33:13.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:12 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:13.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:12 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:13.533 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:13 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[70835]: 2024-04-14T15:33:13.216+0000 7f891790b7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-14T15:33:13.533 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:13 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[70835]: 2024-04-14T15:33:13.248+0000 7f891790b7c0 -1 osd.0 105 log_to_monitors true 2024-04-14T15:33:13.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:13 smithi016 ceph-mon[67348]: from='osd.0 [v2:172.21.15.16:6802/1507114592,v1:172.21.15.16:6803/1507114592]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:33:13.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:13 smithi016 ceph-mon[67348]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:33:13.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:33:13 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[70835]: 2024-04-14T15:33:13.717+0000 7f8908d63700 -1 osd.0 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:33:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:13 smithi016 ceph-mon[64807]: from='osd.0 [v2:172.21.15.16:6802/1507114592,v1:172.21.15.16:6803/1507114592]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:33:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:13 smithi016 ceph-mon[64807]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:33:13.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:13 smithi196 ceph-mon[58679]: from='osd.0 [v2:172.21.15.16:6802/1507114592,v1:172.21.15.16:6803/1507114592]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:33:13.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:13 smithi196 ceph-mon[58679]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T15:33:13.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:33:14.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:14.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[67348]: pgmap v55: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 457 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[67348]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T15:33:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[67348]: osdmap e108: 8 total, 7 up, 8 in 2024-04-14T15:33:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[67348]: from='osd.0 [v2:172.21.15.16:6802/1507114592,v1:172.21.15.16:6803/1507114592]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[67348]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:14.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:14.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[64807]: pgmap v55: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 457 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:14.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[64807]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T15:33:14.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[64807]: osdmap e108: 8 total, 7 up, 8 in 2024-04-14T15:33:14.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[64807]: from='osd.0 [v2:172.21.15.16:6802/1507114592,v1:172.21.15.16:6803/1507114592]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:14.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[64807]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:14.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:14.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:15.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:14 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:15.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:14 smithi196 ceph-mon[58679]: pgmap v55: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 457 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:15.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:14 smithi196 ceph-mon[58679]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T15:33:15.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:14 smithi196 ceph-mon[58679]: osdmap e108: 8 total, 7 up, 8 in 2024-04-14T15:33:15.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:14 smithi196 ceph-mon[58679]: from='osd.0 [v2:172.21.15.16:6802/1507114592,v1:172.21.15.16:6803/1507114592]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:15.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:14 smithi196 ceph-mon[58679]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:15.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:15.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:16.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:15 smithi016 ceph-mon[67348]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:33:16.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:15 smithi016 ceph-mon[67348]: osd.0 [v2:172.21.15.16:6802/1507114592,v1:172.21.15.16:6803/1507114592] boot 2024-04-14T15:33:16.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:15 smithi016 ceph-mon[67348]: osdmap e109: 8 total, 8 up, 8 in 2024-04-14T15:33:16.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:15 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:33:16.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:15 smithi016 ceph-mon[67348]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T15:33:16.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:15 smithi016 ceph-mon[64807]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:33:16.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:15 smithi016 ceph-mon[64807]: osd.0 [v2:172.21.15.16:6802/1507114592,v1:172.21.15.16:6803/1507114592] boot 2024-04-14T15:33:16.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:15 smithi016 ceph-mon[64807]: osdmap e109: 8 total, 8 up, 8 in 2024-04-14T15:33:16.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:15 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:33:16.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:15 smithi016 ceph-mon[64807]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T15:33:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:15 smithi196 ceph-mon[58679]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:33:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:15 smithi196 ceph-mon[58679]: osd.0 [v2:172.21.15.16:6802/1507114592,v1:172.21.15.16:6803/1507114592] boot 2024-04-14T15:33:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:15 smithi196 ceph-mon[58679]: osdmap e109: 8 total, 8 up, 8 in 2024-04-14T15:33:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:15 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T15:33:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:15 smithi196 ceph-mon[58679]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T15:33:17.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:16 smithi016 ceph-mon[67348]: pgmap v58: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:17.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:17.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:17.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:16 smithi016 ceph-mon[64807]: pgmap v58: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:17.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:17.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:17.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:16 smithi196 ceph-mon[58679]: pgmap v58: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:17.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:17.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:18.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:18 smithi196 ceph-mon[58679]: pgmap v60: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:18.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:33:18.935 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:18 smithi016 ceph-mon[67348]: pgmap v60: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:18.935 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:18 smithi016 ceph-mon[64807]: pgmap v60: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-04-14T15:33:20.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:19 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 14. 2024-04-14T15:33:20.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:19 smithi196 systemd[1]: Stopped Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:33:20.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:19 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Start request repeated too quickly. 2024-04-14T15:33:20.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:19 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T15:33:20.165 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Apr 14 15:33:19 smithi196 systemd[1]: Failed to start Ceph grafana.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:33:20.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T15:33:20.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[64807]: Upgrade: osd.1 is safe to restart 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:20.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:20.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T15:33:20.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T15:33:20.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:20 smithi016 ceph-mon[67348]: Upgrade: osd.1 is safe to restart 2024-04-14T15:33:20.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:20.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:20.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:20.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:33:20.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 0 op/s 2024-04-14T15:33:20.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-04-14T15:33:20.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:20.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:33:20.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:20.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:20.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:20.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T15:33:20.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T15:33:20.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:20 smithi196 ceph-mon[58679]: Upgrade: osd.1 is safe to restart 2024-04-14T15:33:21.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:21 smithi016 ceph-mon[67348]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 68/627 objects degraded (10.845%), 20 pgs degraded) 2024-04-14T15:33:21.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:21 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:21.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:21 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:33:21.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:21 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:21.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:21 smithi016 ceph-mon[64807]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 68/627 objects degraded (10.845%), 20 pgs degraded) 2024-04-14T15:33:21.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:21 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:21.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:21 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:33:21.569 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:21 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:21.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:21 smithi196 ceph-mon[58679]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 68/627 objects degraded (10.845%), 20 pgs degraded) 2024-04-14T15:33:21.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:21 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:21.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:21 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T15:33:21.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:21 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:21.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:33:21 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:33:21] "GET /metrics HTTP/1.1" 200 38402 "" "Prometheus/2.33.4" 2024-04-14T15:33:22.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:22 smithi016 ceph-mon[64807]: Upgrade: Updating osd.1 2024-04-14T15:33:22.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:22 smithi016 ceph-mon[64807]: Deploying daemon osd.1 on smithi016 2024-04-14T15:33:22.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:22 smithi016 ceph-mon[64807]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-04-14T15:33:22.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:22 smithi016 ceph-mon[67348]: Upgrade: Updating osd.1 2024-04-14T15:33:22.602 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:22 smithi016 ceph-mon[67348]: Deploying daemon osd.1 on smithi016 2024-04-14T15:33:22.602 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:22 smithi016 ceph-mon[67348]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-04-14T15:33:22.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:22 smithi196 ceph-mon[58679]: Upgrade: Updating osd.1 2024-04-14T15:33:22.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:22 smithi196 ceph-mon[58679]: Deploying daemon osd.1 on smithi016 2024-04-14T15:33:22.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:22 smithi196 ceph-mon[58679]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-04-14T15:33:23.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:33:24.523 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:24 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:24.523 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:24 smithi016 ceph-mon[67348]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T15:33:24.524 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:24 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:24.524 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:24 smithi016 ceph-mon[64807]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T15:33:24.524 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:24 smithi016 systemd[1]: Stopping Ceph osd.1 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:33:24.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:24 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:24.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:24 smithi196 ceph-mon[58679]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T15:33:24.860 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:24 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[35336]: 2024-04-14T15:33:24.522+0000 7fe5fbc6e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:33:24.860 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:24 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[35336]: 2024-04-14T15:33:24.522+0000 7fe5fbc6e700 -1 osd.1 110 *** Got signal Terminated *** 2024-04-14T15:33:24.860 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:24 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[35336]: 2024-04-14T15:33:24.522+0000 7fe5fbc6e700 -1 osd.1 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:33:25.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:25 smithi196 ceph-mon[58679]: osd.1 marked itself down and dead 2024-04-14T15:33:25.708 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:25 smithi016 ceph-mon[67348]: osd.1 marked itself down and dead 2024-04-14T15:33:25.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:25 smithi016 ceph-mon[64807]: osd.1 marked itself down and dead 2024-04-14T15:33:25.999 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:25 smithi016 podman[72722]: 2024-04-14 15:33:25.708262579 +0000 UTC m=+1.286516669 container died c275d1a0eaeb639548c5d1410faae7db94408be864fa0b60131c91218459d201 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1, distribution-scope=public, vendor=Red Hat, Inc., GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.openshift.expose-services=, maintainer=Guillaume Abrioux , name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, version=8, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870) 2024-04-14T15:33:26.360 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:26 smithi016 podman[72722]: 2024-04-14 15:33:26.098221146 +0000 UTC m=+1.676475236 container cleanup c275d1a0eaeb639548c5d1410faae7db94408be864fa0b60131c91218459d201 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, distribution-scope=public, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-14T15:33:26.360 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:26 smithi016 bash[72722]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1 2024-04-14T15:33:26.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:26 smithi196 ceph-mon[58679]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:33:26.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:26 smithi196 ceph-mon[58679]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:33:26.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:26 smithi196 ceph-mon[58679]: osdmap e111: 8 total, 7 up, 8 in 2024-04-14T15:33:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:26 smithi016 ceph-mon[64807]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:33:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:26 smithi016 ceph-mon[64807]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:33:26.709 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:26 smithi016 ceph-mon[64807]: osdmap e111: 8 total, 7 up, 8 in 2024-04-14T15:33:26.709 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:26 smithi016 ceph-mon[67348]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:33:26.709 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:26 smithi016 ceph-mon[67348]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:33:26.710 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:26 smithi016 ceph-mon[67348]: osdmap e111: 8 total, 7 up, 8 in 2024-04-14T15:33:26.710 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:26 smithi016 podman[72739]: 2024-04-14 15:33:26.366092072 +0000 UTC m=+0.652214317 container remove c275d1a0eaeb639548c5d1410faae7db94408be864fa0b60131c91218459d201 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, version=8, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-type=git, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-14T15:33:26.962 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:26 smithi016 podman[72833]: 2024-04-14 15:33:26.612171803 +0000 UTC m=+0.017835982 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:33:26.962 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:26 smithi016 podman[72833]: 2024-04-14 15:33:26.735663795 +0000 UTC m=+0.141327946 container create 1dee07a6e18d7c36355a40091d3fc1e1bc301833b4e651b290c81ce25899ceb0 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-14T15:33:27.264 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:26 smithi016 podman[72833]: 2024-04-14 15:33:26.962245623 +0000 UTC m=+0.367909784 container init 1dee07a6e18d7c36355a40091d3fc1e1bc301833b4e651b290c81ce25899ceb0 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=) 2024-04-14T15:33:27.265 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:26 smithi016 podman[72833]: 2024-04-14 15:33:26.965750136 +0000 UTC m=+0.371414293 container start 1dee07a6e18d7c36355a40091d3fc1e1bc301833b4e651b290c81ce25899ceb0 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403) 2024-04-14T15:33:27.265 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:27 smithi016 podman[72833]: 2024-04-14 15:33:27.036677178 +0000 UTC m=+0.442341332 container attach 1dee07a6e18d7c36355a40091d3fc1e1bc301833b4e651b290c81ce25899ceb0 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True) 2024-04-14T15:33:27.610 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:27 smithi016 podman[72833]: 2024-04-14 15:33:27.265420846 +0000 UTC m=+0.671085012 container died 1dee07a6e18d7c36355a40091d3fc1e1bc301833b4e651b290c81ce25899ceb0 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-14T15:33:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:27 smithi016 ceph-mon[64807]: osdmap e112: 8 total, 7 up, 8 in 2024-04-14T15:33:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:27.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:27 smithi016 ceph-mon[67348]: osdmap e112: 8 total, 7 up, 8 in 2024-04-14T15:33:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:27.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:27 smithi196 ceph-mon[58679]: osdmap e112: 8 total, 7 up, 8 in 2024-04-14T15:33:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:27.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:28.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:27 smithi016 podman[72859]: 2024-04-14 15:33:27.932999266 +0000 UTC m=+0.662367896 container remove 1dee07a6e18d7c36355a40091d3fc1e1bc301833b4e651b290c81ce25899ceb0 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-14T15:33:28.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:27 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-14T15:33:28.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:27 smithi016 systemd[1]: Stopped Ceph osd.1 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:33:28.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:27 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.1.service: Consumed 7.711s CPU time. 2024-04-14T15:33:28.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:28 smithi016 systemd[1]: Starting Ceph osd.1 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:33:28.647 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:28 smithi016 ceph-mon[67348]: pgmap v68: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:33:28.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:28 smithi016 ceph-mon[64807]: pgmap v68: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:33:28.648 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:28 smithi016 podman[72965]: 2024-04-14 15:33:28.367191433 +0000 UTC m=+0.114574835 container create 22eb3cea6e09eeb2af6aaf6c64e800aad9a175c3e63a5e434dfc9f40be446a57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-14T15:33:28.648 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:28 smithi016 podman[72965]: 2024-04-14 15:33:28.270576054 +0000 UTC m=+0.017959456 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:33:28.648 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:28 smithi016 podman[72965]: 2024-04-14 15:33:28.648090564 +0000 UTC m=+0.395473966 container init 22eb3cea6e09eeb2af6aaf6c64e800aad9a175c3e63a5e434dfc9f40be446a57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate, org.label-schema.build-date=20240403, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-14T15:33:28.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:28 smithi196 ceph-mon[58679]: pgmap v68: 161 pgs: 19 stale+active+clean, 142 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:33:28.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:28 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:28.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-14T15:33:28.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:28 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:28.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:33:29.094 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:28 smithi016 podman[72965]: 2024-04-14 15:33:28.651431996 +0000 UTC m=+0.398815408 container start 22eb3cea6e09eeb2af6aaf6c64e800aad9a175c3e63a5e434dfc9f40be446a57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-14T15:33:29.094 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:28 smithi016 podman[72965]: 2024-04-14 15:33:28.761724533 +0000 UTC m=+0.509107940 container attach 22eb3cea6e09eeb2af6aaf6c64e800aad9a175c3e63a5e434dfc9f40be446a57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:33:29.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate[72980]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T15:33:29.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 bash[72965]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T15:33:29.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 bash[72965]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T15:33:29.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 bash[72965]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T15:33:29.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 bash[72965]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T15:33:29.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 bash[72965]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-14T15:33:29.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 bash[72965]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T15:33:29.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 bash[72965]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-14T15:33:29.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate[72980]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T15:33:29.361 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate[72980]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T15:33:29.362 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate[72980]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T15:33:29.362 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate[72980]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-14T15:33:29.362 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate[72980]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T15:33:29.362 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate[72980]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-14T15:33:29.362 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 podman[72965]: 2024-04-14 15:33:29.119037395 +0000 UTC m=+0.866420812 container died 22eb3cea6e09eeb2af6aaf6c64e800aad9a175c3e63a5e434dfc9f40be446a57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-a971fd5) 2024-04-14T15:33:29.718 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:29 smithi196 ceph-mon[58679]: Health check failed: Degraded data redundancy: 88/627 objects degraded (14.035%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:29.838 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:29 smithi016 ceph-mon[67348]: Health check failed: Degraded data redundancy: 88/627 objects degraded (14.035%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:29.839 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:29 smithi016 ceph-mon[64807]: Health check failed: Degraded data redundancy: 88/627 objects degraded (14.035%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:30.111 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:29 smithi016 podman[73101]: 2024-04-14 15:33:29.839124588 +0000 UTC m=+0.714621014 container remove 22eb3cea6e09eeb2af6aaf6c64e800aad9a175c3e63a5e434dfc9f40be446a57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:33:30.111 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:30 smithi016 podman[73165]: 2024-04-14 15:33:30.056128409 +0000 UTC m=+0.112456018 container create 14dec0c76aaf52cc9a21ca47449b4cd27d9629f3d811d58f4095bbc971c6ae96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-14T15:33:30.111 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:30 smithi016 podman[73165]: 2024-04-14 15:33:29.962127129 +0000 UTC m=+0.018454771 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:33:30.427 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:30 smithi016 podman[73165]: 2024-04-14 15:33:30.292331907 +0000 UTC m=+0.348659518 container init 14dec0c76aaf52cc9a21ca47449b4cd27d9629f3d811d58f4095bbc971c6ae96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-14T15:33:30.427 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:30 smithi016 podman[73165]: 2024-04-14 15:33:30.295899546 +0000 UTC m=+0.352227155 container start 14dec0c76aaf52cc9a21ca47449b4cd27d9629f3d811d58f4095bbc971c6ae96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-14T15:33:30.427 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:30 smithi016 ceph-osd[73184]: -- 172.21.15.16:0/2559645697 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561c30441680 con 0x561c3040b000 2024-04-14T15:33:30.427 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:30 smithi016 bash[73165]: 14dec0c76aaf52cc9a21ca47449b4cd27d9629f3d811d58f4095bbc971c6ae96 2024-04-14T15:33:30.427 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:30 smithi016 systemd[1]: Started Ceph osd.1 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:33:30.839 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:30 smithi016 ceph-mon[67348]: pgmap v69: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:30.839 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:30 smithi016 ceph-mon[64807]: pgmap v69: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:30.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:30 smithi196 ceph-mon[58679]: pgmap v69: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:31.111 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[73180]: 2024-04-14T15:33:30.838+0000 7fe639fb47c0 -1 Falling back to public interface 2024-04-14T15:33:31.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:31 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:31.529 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:31 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:31.529 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:31 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:31.529 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:31 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:31.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:33:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:33:31] "GET /metrics HTTP/1.1" 200 38410 "" "Prometheus/2.33.4" 2024-04-14T15:33:31.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:31 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:31.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:31 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:32.844 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:32 smithi016 ceph-mon[67348]: pgmap v70: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:32.881 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:32 smithi016 ceph-mon[64807]: pgmap v70: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:32.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:32 smithi196 ceph-mon[58679]: pgmap v70: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:33.861 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[73180]: 2024-04-14T15:33:33.569+0000 7fe639fb47c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-14T15:33:33.861 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:33 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[73180]: 2024-04-14T15:33:33.595+0000 7fe639fb47c0 -1 osd.1 110 log_to_monitors true 2024-04-14T15:33:33.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:33:34.721 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:33:34 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[73180]: 2024-04-14T15:33:34.485+0000 7fe62b40c700 -1 osd.1 110 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:33:34.722 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:34 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:34.722 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:34 smithi016 ceph-mon[67348]: pgmap v71: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:34.722 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:34 smithi016 ceph-mon[67348]: from='osd.1 [v2:172.21.15.16:6810/590894843,v1:172.21.15.16:6811/590894843]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:33:34.722 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:34 smithi016 ceph-mon[67348]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:33:34.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:34 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:34.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:34 smithi016 ceph-mon[64807]: pgmap v71: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:34.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:34 smithi016 ceph-mon[64807]: from='osd.1 [v2:172.21.15.16:6810/590894843,v1:172.21.15.16:6811/590894843]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:33:34.723 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:34 smithi016 ceph-mon[64807]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:33:34.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:34 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:34 smithi196 ceph-mon[58679]: pgmap v71: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:34 smithi196 ceph-mon[58679]: from='osd.1 [v2:172.21.15.16:6810/590894843,v1:172.21.15.16:6811/590894843]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:33:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:34 smithi196 ceph-mon[58679]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T15:33:35.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[67348]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T15:33:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[67348]: osdmap e113: 8 total, 7 up, 8 in 2024-04-14T15:33:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[67348]: from='osd.1 [v2:172.21.15.16:6810/590894843,v1:172.21.15.16:6811/590894843]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[67348]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:35.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[64807]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T15:33:35.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[64807]: osdmap e113: 8 total, 7 up, 8 in 2024-04-14T15:33:35.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[64807]: from='osd.1 [v2:172.21.15.16:6810/590894843,v1:172.21.15.16:6811/590894843]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:35.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[64807]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:35.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:35.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:35 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:35 smithi196 ceph-mon[58679]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T15:33:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:35 smithi196 ceph-mon[58679]: osdmap e113: 8 total, 7 up, 8 in 2024-04-14T15:33:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:35 smithi196 ceph-mon[58679]: from='osd.1 [v2:172.21.15.16:6810/590894843,v1:172.21.15.16:6811/590894843]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:35 smithi196 ceph-mon[58679]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:35 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:35 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:36.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[67348]: pgmap v73: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:36.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[67348]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:33:36.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[67348]: osd.1 [v2:172.21.15.16:6810/590894843,v1:172.21.15.16:6811/590894843] boot 2024-04-14T15:33:36.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[67348]: osdmap e114: 8 total, 8 up, 8 in 2024-04-14T15:33:36.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:33:36.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:36.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:36.735 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[67348]: osdmap e115: 8 total, 8 up, 8 in 2024-04-14T15:33:36.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[64807]: pgmap v73: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:36.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[64807]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:33:36.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[64807]: osd.1 [v2:172.21.15.16:6810/590894843,v1:172.21.15.16:6811/590894843] boot 2024-04-14T15:33:36.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[64807]: osdmap e114: 8 total, 8 up, 8 in 2024-04-14T15:33:36.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:33:36.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:36.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:36.736 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:36 smithi016 ceph-mon[64807]: osdmap e115: 8 total, 8 up, 8 in 2024-04-14T15:33:36.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:36 smithi196 ceph-mon[58679]: pgmap v73: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 88/627 objects degraded (14.035%) 2024-04-14T15:33:36.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:36 smithi196 ceph-mon[58679]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:33:36.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:36 smithi196 ceph-mon[58679]: osd.1 [v2:172.21.15.16:6810/590894843,v1:172.21.15.16:6811/590894843] boot 2024-04-14T15:33:36.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:36 smithi196 ceph-mon[58679]: osdmap e114: 8 total, 8 up, 8 in 2024-04-14T15:33:36.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:36 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T15:33:36.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:36 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:36.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:36 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:36.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:36 smithi196 ceph-mon[58679]: osdmap e115: 8 total, 8 up, 8 in 2024-04-14T15:33:38.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:38 smithi016 ceph-mon[64807]: pgmap v76: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 88/627 objects degraded (14.035%) 2024-04-14T15:33:38.614 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:38 smithi016 ceph-mon[67348]: pgmap v76: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 88/627 objects degraded (14.035%) 2024-04-14T15:33:38.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:38 smithi196 ceph-mon[58679]: pgmap v76: 161 pgs: 25 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 129 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 88/627 objects degraded (14.035%) 2024-04-14T15:33:38.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-14T15:33:38.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:33:39.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:39 smithi196 ceph-mon[58679]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:39.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:39 smithi016 ceph-mon[67348]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:39 smithi016 ceph-mon[64807]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:40.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: pgmap v77: 161 pgs: 6 active+undersized+degraded, 13 active+undersized, 142 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-04-14T15:33:40.768 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:33:40.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:33:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-04-14T15:33:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:33:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T15:33:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T15:33:40.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[64807]: Upgrade: osd.2 is safe to restart 2024-04-14T15:33:40.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: pgmap v77: 161 pgs: 6 active+undersized+degraded, 13 active+undersized, 142 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-04-14T15:33:40.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:40.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:40.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:40.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:33:40.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-04-14T15:33:40.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:40.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:40.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:33:40.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:40.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:40.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:40.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T15:33:40.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T15:33:40.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:40 smithi016 ceph-mon[67348]: Upgrade: osd.2 is safe to restart 2024-04-14T15:33:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: pgmap v77: 161 pgs: 6 active+undersized+degraded, 13 active+undersized, 142 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-04-14T15:33:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:33:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-04-14T15:33:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:33:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:40.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:40.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:40.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T15:33:40.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T15:33:40.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:40 smithi196 ceph-mon[58679]: Upgrade: osd.2 is safe to restart 2024-04-14T15:33:41.461 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:33:41.461 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (3m) 6s ago 8m 23.9M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:33:41.461 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 unknown 76s ago 8m - - 2024-04-14T15:33:41.461 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (2m) 6s ago 7m 52.1M - 3.5 e1d6a67b021e 27d95fa4403f 2024-04-14T15:33:41.462 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (2m) 76s ago 12m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:33:41.462 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (3m) 6s ago 14m 505M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:33:41.462 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (2m) 6s ago 14m 46.2M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:33:41.462 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (86s) 76s ago 13m 22.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 72817f80f666 2024-04-14T15:33:41.462 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (103s) 6s ago 13m 37.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:33:41.462 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (3m) 6s ago 9m 20.3M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:33:41.462 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (3m) 76s ago 9m 18.0M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:33:41.462 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (31s) 6s ago 12m 42.2M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 f6ee7262903f 2024-04-14T15:33:41.462 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (11s) 6s ago 12m 12.5M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 14dec0c76aaf 2024-04-14T15:33:41.462 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (11m) 6s ago 11m 53.3M 2737M 17.2.0 e1d6a67b021e 592ce301a74a 2024-04-14T15:33:41.463 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (11m) 6s ago 11m 53.2M 2737M 17.2.0 e1d6a67b021e 1ed8730c3fc1 2024-04-14T15:33:41.463 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (11m) 76s ago 11m 54.4M 2992M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:33:41.463 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (10m) 76s ago 10m 50.8M 2992M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:33:41.463 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (10m) 76s ago 10m 49.6M 2992M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:33:41.463 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (10m) 76s ago 9m 54.7M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:33:41.463 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (2m) 76s ago 9m 71.0M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:33:41.463 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (7m) 6s ago 7m 75.1M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:33:41.463 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (7m) 76s ago 7m 74.9M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:33:41.463 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (7m) 6s ago 7m 73.0M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:33:41.463 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (7m) 76s ago 7m 75.3M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:33:41.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:33:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:33:41] "GET /metrics HTTP/1.1" 200 38410 "" "Prometheus/2.33.4" 2024-04-14T15:33:41.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:41 smithi016 ceph-mon[67348]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded) 2024-04-14T15:33:41.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:41 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:41.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:41 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:33:41.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:41 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:41.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:41 smithi016 ceph-mon[64807]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded) 2024-04-14T15:33:41.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:41 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:41.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:41 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:33:41.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:41 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:33:41.893 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:33:41.894 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:33:41.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:33:41.894 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:33:41.894 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-14T15:33:41.894 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 7 2024-04-14T15:33:41.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:33:41.894 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:33:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:41 smithi196 ceph-mon[58679]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded) 2024-04-14T15:33:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:41 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:41 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T15:33:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:41 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:33:42.246 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:33:42.247 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:33:42.247 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:33:42.247 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:33:42.247 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:33:42.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-04-14T15:33:42.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-14T15:33:42.247 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:33:42.247 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "7/23 daemons upgraded", 2024-04-14T15:33:42.247 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T15:33:42.247 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:33:42.248 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:33:42.665 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:33:42.665 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:33:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:33:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[67348]: from='client.54223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[67348]: from='client.44262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[67348]: Upgrade: Updating osd.2 2024-04-14T15:33:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[67348]: Deploying daemon osd.2 on smithi016 2024-04-14T15:33:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[67348]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[67348]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 836 B/s rd, 0 op/s 2024-04-14T15:33:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/2515160086' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[64807]: from='client.54223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[64807]: from='client.44262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[64807]: Upgrade: Updating osd.2 2024-04-14T15:33:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[64807]: Deploying daemon osd.2 on smithi016 2024-04-14T15:33:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[64807]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[64807]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 836 B/s rd, 0 op/s 2024-04-14T15:33:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:42 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/2515160086' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:42 smithi196 ceph-mon[58679]: from='client.54223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:42 smithi196 ceph-mon[58679]: from='client.44262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:42 smithi196 ceph-mon[58679]: Upgrade: Updating osd.2 2024-04-14T15:33:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:42 smithi196 ceph-mon[58679]: Deploying daemon osd.2 on smithi016 2024-04-14T15:33:42.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:42 smithi196 ceph-mon[58679]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:42.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:42 smithi196 ceph-mon[58679]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 836 B/s rd, 0 op/s 2024-04-14T15:33:42.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:42.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:42.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:42 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/2515160086' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:33:43.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:43 smithi016 ceph-mon[67348]: from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:43.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:43 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/197026537' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:33:43.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:43 smithi016 ceph-mon[64807]: from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:43.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:43 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/197026537' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:33:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:43 smithi196 ceph-mon[58679]: from='client.54247 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:33:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:43 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/197026537' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:33:43.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:33:44.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:44 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:44.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:44 smithi016 ceph-mon[67348]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:33:44.611 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:44 smithi016 systemd[1]: Stopping Ceph osd.2 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:33:44.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:44 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:44.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:44 smithi016 ceph-mon[64807]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:33:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:44 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:44 smithi196 ceph-mon[58679]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:33:45.110 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[38251]: 2024-04-14T15:33:44.789+0000 7f1ca8c01700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:33:45.111 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[38251]: 2024-04-14T15:33:44.789+0000 7f1ca8c01700 -1 osd.2 115 *** Got signal Terminated *** 2024-04-14T15:33:45.111 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[38251]: 2024-04-14T15:33:44.789+0000 7f1ca8c01700 -1 osd.2 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:33:45.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:45 smithi016 ceph-mon[67348]: osd.2 marked itself down and dead 2024-04-14T15:33:45.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:45 smithi016 ceph-mon[64807]: osd.2 marked itself down and dead 2024-04-14T15:33:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:45 smithi196 ceph-mon[58679]: osd.2 marked itself down and dead 2024-04-14T15:33:46.308 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:46 smithi016 podman[75187]: 2024-04-14 15:33:46.027037109 +0000 UTC m=+1.341423260 container died 592ce301a74a96523fcf06da562ed7d37796de3d9e76fc76ed5ec9e2c5fec822 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.tags=base centos centos-stream, release=754, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, RELEASE=HEAD, io.buildah.version=1.19.8, version=8, com.redhat.component=centos-stream-container) 2024-04-14T15:33:46.578 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:46 smithi016 podman[75187]: 2024-04-14 15:33:46.406878038 +0000 UTC m=+1.721264184 container cleanup 592ce301a74a96523fcf06da562ed7d37796de3d9e76fc76ed5ec9e2c5fec822 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, io.openshift.expose-services=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, vcs-type=git, architecture=x86_64, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-14T15:33:46.579 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:46 smithi016 bash[75187]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2 2024-04-14T15:33:46.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:46 smithi016 ceph-mon[67348]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:46.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:46 smithi016 ceph-mon[67348]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:33:46.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:46 smithi016 ceph-mon[67348]: osdmap e116: 8 total, 7 up, 8 in 2024-04-14T15:33:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:46 smithi016 ceph-mon[64807]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:46 smithi016 ceph-mon[64807]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:33:46.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:46 smithi016 ceph-mon[64807]: osdmap e116: 8 total, 7 up, 8 in 2024-04-14T15:33:46.861 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:46 smithi016 podman[75204]: 2024-04-14 15:33:46.675136862 +0000 UTC m=+0.648658399 container remove 592ce301a74a96523fcf06da562ed7d37796de3d9e76fc76ed5ec9e2c5fec822 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_CLEAN=True, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_BRANCH=HEAD, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc.) 2024-04-14T15:33:46.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:46 smithi196 ceph-mon[58679]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:33:46.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:46 smithi196 ceph-mon[58679]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:33:46.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:46 smithi196 ceph-mon[58679]: osdmap e116: 8 total, 7 up, 8 in 2024-04-14T15:33:47.345 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:47 smithi016 podman[75297]: 2024-04-14 15:33:46.933085177 +0000 UTC m=+0.019373991 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:33:47.345 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:47 smithi016 podman[75297]: 2024-04-14 15:33:47.034329013 +0000 UTC m=+0.120617848 container create 64d310a97c16ce797f93b5fbbec46bb75d7719a59d75049ab7e7e3747f31f614 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-14T15:33:47.345 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:47 smithi016 podman[75297]: 2024-04-14 15:33:47.261867616 +0000 UTC m=+0.348156403 container init 64d310a97c16ce797f93b5fbbec46bb75d7719a59d75049ab7e7e3747f31f614 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux ) 2024-04-14T15:33:47.345 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:47 smithi016 podman[75297]: 2024-04-14 15:33:47.266123464 +0000 UTC m=+0.352412252 container start 64d310a97c16ce797f93b5fbbec46bb75d7719a59d75049ab7e7e3747f31f614 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:33:47.345 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:47 smithi016 podman[75297]: 2024-04-14 15:33:47.345142703 +0000 UTC m=+0.431431490 container attach 64d310a97c16ce797f93b5fbbec46bb75d7719a59d75049ab7e7e3747f31f614 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:33:47.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:47 smithi016 ceph-mon[67348]: osdmap e117: 8 total, 7 up, 8 in 2024-04-14T15:33:47.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:47 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:47 smithi016 ceph-mon[64807]: osdmap e117: 8 total, 7 up, 8 in 2024-04-14T15:33:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:47 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:47.611 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:47 smithi016 podman[75297]: 2024-04-14 15:33:47.570915976 +0000 UTC m=+0.657204777 container died 64d310a97c16ce797f93b5fbbec46bb75d7719a59d75049ab7e7e3747f31f614 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS) 2024-04-14T15:33:47.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:47 smithi196 ceph-mon[58679]: osdmap e117: 8 total, 7 up, 8 in 2024-04-14T15:33:47.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:47 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:48.587 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:48 smithi016 podman[75323]: 2024-04-14 15:33:48.274784917 +0000 UTC m=+0.699181427 container remove 64d310a97c16ce797f93b5fbbec46bb75d7719a59d75049ab7e7e3747f31f614 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:33:48.587 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:48 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-14T15:33:48.587 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:48 smithi016 systemd[1]: Stopped Ceph osd.2 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:33:48.587 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:48 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.2.service: Consumed 6.203s CPU time. 2024-04-14T15:33:48.587 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:48 smithi016 systemd[1]: Starting Ceph osd.2 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:33:48.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:48 smithi016 ceph-mon[67348]: pgmap v85: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:33:48.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:48 smithi016 ceph-mon[64807]: pgmap v85: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:33:48.861 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:48 smithi016 podman[75429]: 2024-04-14 15:33:48.634953249 +0000 UTC m=+0.017985939 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:33:48.861 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:48 smithi016 podman[75429]: 2024-04-14 15:33:48.742754386 +0000 UTC m=+0.125787098 container create 80f56f3eb368001fb42c54dd90d6c43598c27bcc8d5da77b36f97db5d8361955 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-14T15:33:48.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:48 smithi196 ceph-mon[58679]: pgmap v85: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T15:33:48.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:48.660Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-14T15:33:48.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:48.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:33:49.361 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:48 smithi016 podman[75429]: 2024-04-14 15:33:48.987197369 +0000 UTC m=+0.370230040 container init 80f56f3eb368001fb42c54dd90d6c43598c27bcc8d5da77b36f97db5d8361955 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:33:49.361 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:48 smithi016 podman[75429]: 2024-04-14 15:33:48.990378718 +0000 UTC m=+0.373411395 container start 80f56f3eb368001fb42c54dd90d6c43598c27bcc8d5da77b36f97db5d8361955 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-14T15:33:49.361 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 podman[75429]: 2024-04-14 15:33:49.070262722 +0000 UTC m=+0.453295405 container attach 80f56f3eb368001fb42c54dd90d6c43598c27bcc8d5da77b36f97db5d8361955 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-04-14T15:33:49.832 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:49 smithi016 ceph-mon[67348]: Health check failed: Degraded data redundancy: 54/627 objects degraded (8.612%), 12 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:49.833 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:49 smithi016 ceph-mon[64807]: Health check failed: Degraded data redundancy: 54/627 objects degraded (8.612%), 12 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:49.833 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate[75445]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T15:33:49.833 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate[75445]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T15:33:49.833 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate[75445]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T15:33:49.833 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate[75445]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T15:33:49.833 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate[75445]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-14T15:33:49.834 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate[75445]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T15:33:49.834 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate[75445]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-14T15:33:49.834 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 bash[75429]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T15:33:49.834 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 bash[75429]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T15:33:49.834 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 bash[75429]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T15:33:49.834 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 bash[75429]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T15:33:49.834 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 bash[75429]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-14T15:33:49.834 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 bash[75429]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T15:33:49.834 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 bash[75429]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-14T15:33:49.834 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:49 smithi016 podman[75429]: 2024-04-14 15:33:49.469969068 +0000 UTC m=+0.853001758 container died 80f56f3eb368001fb42c54dd90d6c43598c27bcc8d5da77b36f97db5d8361955 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-14T15:33:49.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:49 smithi196 ceph-mon[58679]: Health check failed: Degraded data redundancy: 54/627 objects degraded (8.612%), 12 pgs degraded (PG_DEGRADED) 2024-04-14T15:33:50.466 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:50 smithi016 podman[75578]: 2024-04-14 15:33:50.116762257 +0000 UTC m=+0.641815169 container remove 80f56f3eb368001fb42c54dd90d6c43598c27bcc8d5da77b36f97db5d8361955 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-14T15:33:50.467 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:50 smithi016 podman[75643]: 2024-04-14 15:33:50.23568174 +0000 UTC m=+0.017676470 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:33:50.467 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:50 smithi016 podman[75643]: 2024-04-14 15:33:50.342617165 +0000 UTC m=+0.124611885 container create ce205c8399d32447883d67226dece43142090246c75e72b93839619d42a996fc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-14T15:33:50.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:50 smithi196 ceph-mon[58679]: pgmap v86: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:51.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:50 smithi016 ceph-mon[64807]: pgmap v86: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:51.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:50 smithi016 ceph-mon[67348]: pgmap v86: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:51.111 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:50 smithi016 podman[75643]: 2024-04-14 15:33:50.611611794 +0000 UTC m=+0.393606514 container init ce205c8399d32447883d67226dece43142090246c75e72b93839619d42a996fc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5) 2024-04-14T15:33:51.111 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:50 smithi016 podman[75643]: 2024-04-14 15:33:50.614752958 +0000 UTC m=+0.396747679 container start ce205c8399d32447883d67226dece43142090246c75e72b93839619d42a996fc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:33:51.111 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:50 smithi016 ceph-osd[75661]: -- 172.21.15.16:0/2961242813 <== mon.0 v2:172.21.15.16:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bf987ff680 con 0x55bf987c9000 2024-04-14T15:33:51.111 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:50 smithi016 bash[75643]: ce205c8399d32447883d67226dece43142090246c75e72b93839619d42a996fc 2024-04-14T15:33:51.111 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:50 smithi016 systemd[1]: Started Ceph osd.2 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:33:51.566 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:51 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[75657]: 2024-04-14T15:33:51.158+0000 7f3c7d9f47c0 -1 Falling back to public interface 2024-04-14T15:33:51.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:33:51 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:33:51] "GET /metrics HTTP/1.1" 200 38413 "" "Prometheus/2.33.4" 2024-04-14T15:33:51.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:51 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:51.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:51 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:51.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:51 smithi016 ceph-mon[67348]: pgmap v87: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 54/627 objects degraded (8.612%) 2024-04-14T15:33:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:51 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:51 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:51 smithi016 ceph-mon[64807]: pgmap v87: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 54/627 objects degraded (8.612%) 2024-04-14T15:33:52.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:51 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:52.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:51 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:52.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:51 smithi196 ceph-mon[58679]: pgmap v87: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 54/627 objects degraded (8.612%) 2024-04-14T15:33:53.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:53 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:33:54.134 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:53 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[75657]: 2024-04-14T15:33:53.874+0000 7f3c7d9f47c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-14T15:33:54.134 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:53 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[75657]: 2024-04-14T15:33:53.896+0000 7f3c7d9f47c0 -1 osd.2 115 log_to_monitors true 2024-04-14T15:33:54.772 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:54 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:54.773 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:54 smithi016 ceph-mon[67348]: pgmap v88: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:54.773 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:54 smithi016 ceph-mon[67348]: from='osd.2 [v2:172.21.15.16:6818/4086181321,v1:172.21.15.16:6819/4086181321]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:33:54.773 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:54 smithi016 ceph-mon[67348]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:33:54.773 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:54 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:54.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:54 smithi016 ceph-mon[64807]: pgmap v88: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:54.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:54 smithi016 ceph-mon[64807]: from='osd.2 [v2:172.21.15.16:6818/4086181321,v1:172.21.15.16:6819/4086181321]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:33:54.774 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:54 smithi016 ceph-mon[64807]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:33:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:54 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:33:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:54 smithi196 ceph-mon[58679]: pgmap v88: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:54 smithi196 ceph-mon[58679]: from='osd.2 [v2:172.21.15.16:6818/4086181321,v1:172.21.15.16:6819/4086181321]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:33:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:54 smithi196 ceph-mon[58679]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T15:33:55.110 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:33:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[75657]: 2024-04-14T15:33:54.919+0000 7f3c6ae44700 -1 osd.2 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:33:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:55 smithi196 ceph-mon[58679]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T15:33:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:55 smithi196 ceph-mon[58679]: osdmap e118: 8 total, 7 up, 8 in 2024-04-14T15:33:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:55 smithi196 ceph-mon[58679]: from='osd.2 [v2:172.21.15.16:6818/4086181321,v1:172.21.15.16:6819/4086181321]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:55 smithi196 ceph-mon[58679]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:55 smithi196 ceph-mon[58679]: from='osd.2 ' entity='osd.2' 2024-04-14T15:33:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:55 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:55 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:56.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[67348]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T15:33:56.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[67348]: osdmap e118: 8 total, 7 up, 8 in 2024-04-14T15:33:56.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[67348]: from='osd.2 [v2:172.21.15.16:6818/4086181321,v1:172.21.15.16:6819/4086181321]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:56.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[67348]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:56.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[67348]: from='osd.2 ' entity='osd.2' 2024-04-14T15:33:56.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:56.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[64807]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T15:33:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[64807]: osdmap e118: 8 total, 7 up, 8 in 2024-04-14T15:33:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[64807]: from='osd.2 [v2:172.21.15.16:6818/4086181321,v1:172.21.15.16:6819/4086181321]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:56.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[64807]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:33:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[64807]: from='osd.2 ' entity='osd.2' 2024-04-14T15:33:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:56.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:55 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:56.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:56 smithi196 ceph-mon[58679]: pgmap v90: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 113 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:56.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:56 smithi196 ceph-mon[58679]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:33:56.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:56 smithi196 ceph-mon[58679]: osd.2 [v2:172.21.15.16:6818/4086181321,v1:172.21.15.16:6819/4086181321] boot 2024-04-14T15:33:56.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:56 smithi196 ceph-mon[58679]: osdmap e119: 8 total, 8 up, 8 in 2024-04-14T15:33:56.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:56 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:33:56.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:56 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:56.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:56 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:57.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[67348]: pgmap v90: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 113 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:57.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[67348]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:33:57.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[67348]: osd.2 [v2:172.21.15.16:6818/4086181321,v1:172.21.15.16:6819/4086181321] boot 2024-04-14T15:33:57.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[67348]: osdmap e119: 8 total, 8 up, 8 in 2024-04-14T15:33:57.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:33:57.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:57.099 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:57.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[64807]: pgmap v90: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 113 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:57.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[64807]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:33:57.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[64807]: osd.2 [v2:172.21.15.16:6818/4086181321,v1:172.21.15.16:6819/4086181321] boot 2024-04-14T15:33:57.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[64807]: osdmap e119: 8 total, 8 up, 8 in 2024-04-14T15:33:57.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T15:33:57.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:57.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:56 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:58.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:58.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:58.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:57 smithi196 ceph-mon[58679]: osdmap e120: 8 total, 8 up, 8 in 2024-04-14T15:33:58.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:57 smithi196 ceph-mon[58679]: pgmap v93: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:58.232 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:58.233 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:58.233 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:57 smithi016 ceph-mon[67348]: osdmap e120: 8 total, 8 up, 8 in 2024-04-14T15:33:58.233 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:57 smithi016 ceph-mon[67348]: pgmap v93: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:58.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:33:58.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:33:58.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:57 smithi016 ceph-mon[64807]: osdmap e120: 8 total, 8 up, 8 in 2024-04-14T15:33:58.233 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:57 smithi016 ceph-mon[64807]: pgmap v93: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 146 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T15:33:58.914 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:33:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:33:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:00.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:33:59 smithi016 ceph-mon[67348]: Health check update: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:00.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:33:59 smithi016 ceph-mon[64807]: Health check update: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:00.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:33:59 smithi196 ceph-mon[58679]: Health check update: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[64807]: pgmap v94: 161 pgs: 2 active+undersized+degraded, 10 active+undersized, 149 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-04-14T15:34:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:00.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:00.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:34:00.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:00.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:34:00.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:00.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:00.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:00.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T15:34:00.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[67348]: pgmap v94: 161 pgs: 2 active+undersized+degraded, 10 active+undersized, 149 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-04-14T15:34:00.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:00.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:00.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:00.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:34:00.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:00.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:34:00.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:00.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:00.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:00.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:00 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T15:34:01.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:00 smithi196 ceph-mon[58679]: pgmap v94: 161 pgs: 2 active+undersized+degraded, 10 active+undersized, 149 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-04-14T15:34:01.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:01.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:01.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:01.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:34:01.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:01.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:34:01.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:01.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:01.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:01.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:00 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T15:34:01.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:34:01] "GET /metrics HTTP/1.1" 200 38293 "" "Prometheus/2.33.4" 2024-04-14T15:34:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[67348]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-04-14T15:34:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[67348]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[67348]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T15:34:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[67348]: Upgrade: osd.3 is safe to restart 2024-04-14T15:34:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[67348]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded) 2024-04-14T15:34:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:34:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:01.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[64807]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-04-14T15:34:01.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[64807]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:01.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[64807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T15:34:01.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[64807]: Upgrade: osd.3 is safe to restart 2024-04-14T15:34:01.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[64807]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded) 2024-04-14T15:34:01.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:01.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:34:01.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:01 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:02.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:01 smithi196 ceph-mon[58679]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-04-14T15:34:02.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:01 smithi196 ceph-mon[58679]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:02.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:01 smithi196 ceph-mon[58679]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T15:34:02.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:01 smithi196 ceph-mon[58679]: Upgrade: osd.3 is safe to restart 2024-04-14T15:34:02.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:01 smithi196 ceph-mon[58679]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded) 2024-04-14T15:34:02.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:01 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:02.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:01 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T15:34:02.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:01 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:02.855 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:02 smithi016 ceph-mon[67348]: Upgrade: Updating osd.3 2024-04-14T15:34:02.855 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:02 smithi016 ceph-mon[67348]: Deploying daemon osd.3 on smithi016 2024-04-14T15:34:02.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:02 smithi016 ceph-mon[64807]: Upgrade: Updating osd.3 2024-04-14T15:34:02.855 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:02 smithi016 ceph-mon[64807]: Deploying daemon osd.3 on smithi016 2024-04-14T15:34:03.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:02 smithi196 ceph-mon[58679]: Upgrade: Updating osd.3 2024-04-14T15:34:03.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:02 smithi196 ceph-mon[58679]: Deploying daemon osd.3 on smithi016 2024-04-14T15:34:03.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:03 smithi196 ceph-mon[58679]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-04-14T15:34:03.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:04.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:03 smithi016 ceph-mon[67348]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-04-14T15:34:04.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:03 smithi016 ceph-mon[64807]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-04-14T15:34:05.034 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:04 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:05.035 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:04 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:05.035 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:04 smithi016 systemd[1]: Stopping Ceph osd.3 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:34:05.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:04 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:05.360 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:05 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[41211]: 2024-04-14T15:34:05.033+0000 7fd4194d2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:34:05.361 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:05 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[41211]: 2024-04-14T15:34:05.033+0000 7fd4194d2700 -1 osd.3 120 *** Got signal Terminated *** 2024-04-14T15:34:05.361 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:05 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[41211]: 2024-04-14T15:34:05.033+0000 7fd4194d2700 -1 osd.3 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:34:06.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:05 smithi016 ceph-mon[67348]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:34:06.033 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:05 smithi016 ceph-mon[67348]: osd.3 marked itself down and dead 2024-04-14T15:34:06.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:05 smithi016 ceph-mon[64807]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:34:06.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:05 smithi016 ceph-mon[64807]: osd.3 marked itself down and dead 2024-04-14T15:34:06.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:05 smithi196 ceph-mon[58679]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:34:06.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:05 smithi196 ceph-mon[58679]: osd.3 marked itself down and dead 2024-04-14T15:34:06.293 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:06 smithi016 podman[77526]: 2024-04-14 15:34:06.033734325 +0000 UTC m=+1.103078264 container died 1ed8730c3fc16efd81266edf384d13e3752c017c0ff131ecbd11820ec3eff5fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3, name=centos-stream, distribution-scope=public, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, com.redhat.component=centos-stream-container, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-14T15:34:06.610 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:06 smithi016 podman[77526]: 2024-04-14 15:34:06.392374808 +0000 UTC m=+1.461718747 container cleanup 1ed8730c3fc16efd81266edf384d13e3752c017c0ff131ecbd11820ec3eff5fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3, distribution-scope=public, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_CLEAN=True, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, name=centos-stream, RELEASE=HEAD, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, ceph=True) 2024-04-14T15:34:06.611 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:06 smithi016 bash[77526]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3 2024-04-14T15:34:07.045 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:06 smithi016 ceph-mon[67348]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:34:07.045 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:06 smithi016 ceph-mon[67348]: osdmap e121: 8 total, 7 up, 8 in 2024-04-14T15:34:07.045 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:06 smithi016 ceph-mon[64807]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:34:07.045 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:06 smithi016 ceph-mon[64807]: osdmap e121: 8 total, 7 up, 8 in 2024-04-14T15:34:07.045 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:06 smithi016 podman[77543]: 2024-04-14 15:34:06.719159855 +0000 UTC m=+0.683237007 container remove 1ed8730c3fc16efd81266edf384d13e3752c017c0ff131ecbd11820ec3eff5fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.openshift.expose-services=, distribution-scope=public, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_CLEAN=True, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, ceph=True, GIT_BRANCH=HEAD) 2024-04-14T15:34:07.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:06 smithi196 ceph-mon[58679]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:34:07.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:06 smithi196 ceph-mon[58679]: osdmap e121: 8 total, 7 up, 8 in 2024-04-14T15:34:07.360 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:07 smithi016 podman[77627]: 2024-04-14 15:34:07.04512572 +0000 UTC m=+0.101380602 container create 0b0d5203f3128a1ace6cfc9790f5bea47125acd517d0032e6e200c4e37718410 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-14T15:34:07.361 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:07 smithi016 podman[77627]: 2024-04-14 15:34:06.961636475 +0000 UTC m=+0.017891317 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:34:07.667 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:07 smithi016 podman[77627]: 2024-04-14 15:34:07.364440746 +0000 UTC m=+0.420695601 container init 0b0d5203f3128a1ace6cfc9790f5bea47125acd517d0032e6e200c4e37718410 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:34:07.667 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:07 smithi016 podman[77627]: 2024-04-14 15:34:07.367717758 +0000 UTC m=+0.423972603 container start 0b0d5203f3128a1ace6cfc9790f5bea47125acd517d0032e6e200c4e37718410 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:34:07.667 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:07 smithi016 podman[77627]: 2024-04-14 15:34:07.464380548 +0000 UTC m=+0.520635396 container attach 0b0d5203f3128a1ace6cfc9790f5bea47125acd517d0032e6e200c4e37718410 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-14T15:34:08.026 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:07 smithi016 ceph-mon[67348]: pgmap v100: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-14T15:34:08.026 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:07 smithi016 ceph-mon[67348]: osdmap e122: 8 total, 7 up, 8 in 2024-04-14T15:34:08.027 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:07 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:08.027 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:07 smithi016 podman[77627]: 2024-04-14 15:34:07.667609503 +0000 UTC m=+0.723864336 container died 0b0d5203f3128a1ace6cfc9790f5bea47125acd517d0032e6e200c4e37718410 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5) 2024-04-14T15:34:08.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:07 smithi016 ceph-mon[64807]: pgmap v100: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-14T15:34:08.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:07 smithi016 ceph-mon[64807]: osdmap e122: 8 total, 7 up, 8 in 2024-04-14T15:34:08.027 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:07 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:08.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:07 smithi196 ceph-mon[58679]: pgmap v100: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-14T15:34:08.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:07 smithi196 ceph-mon[58679]: osdmap e122: 8 total, 7 up, 8 in 2024-04-14T15:34:08.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:07 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:08.611 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:08 smithi016 podman[77653]: 2024-04-14 15:34:08.302140927 +0000 UTC m=+0.626397933 container remove 0b0d5203f3128a1ace6cfc9790f5bea47125acd517d0032e6e200c4e37718410 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-14T15:34:08.611 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:08 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-14T15:34:08.611 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:08 smithi016 systemd[1]: Stopped Ceph osd.3 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:34:08.611 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:08 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.3.service: Consumed 8.049s CPU time. 2024-04-14T15:34:08.611 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:08 smithi016 systemd[1]: Starting Ceph osd.3 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:34:08.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:08 smithi196 ceph-mon[58679]: pgmap v102: 161 pgs: 13 active+undersized+degraded, 13 stale+active+clean, 20 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-04-14T15:34:08.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:08 smithi196 ceph-mon[58679]: Health check failed: Degraded data redundancy: 38/627 objects degraded (6.061%), 13 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:08.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:09.073 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:08 smithi016 ceph-mon[67348]: pgmap v102: 161 pgs: 13 active+undersized+degraded, 13 stale+active+clean, 20 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-04-14T15:34:09.073 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:08 smithi016 ceph-mon[67348]: Health check failed: Degraded data redundancy: 38/627 objects degraded (6.061%), 13 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:09.073 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:08 smithi016 ceph-mon[64807]: pgmap v102: 161 pgs: 13 active+undersized+degraded, 13 stale+active+clean, 20 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 38/627 objects degraded (6.061%) 2024-04-14T15:34:09.073 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:08 smithi016 ceph-mon[64807]: Health check failed: Degraded data redundancy: 38/627 objects degraded (6.061%), 13 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:09.073 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:08 smithi016 podman[77755]: 2024-04-14 15:34:08.654730855 +0000 UTC m=+0.017913742 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:34:09.074 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:08 smithi016 podman[77755]: 2024-04-14 15:34:08.753521875 +0000 UTC m=+0.116704756 container create be57ad3b61b917f7060108d433c4e1414f77e861892c01c82d010b98bbeb7b96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5) 2024-04-14T15:34:09.074 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:08 smithi016 podman[77755]: 2024-04-14 15:34:08.998387594 +0000 UTC m=+0.361570466 container init be57ad3b61b917f7060108d433c4e1414f77e861892c01c82d010b98bbeb7b96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-14T15:34:09.074 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 podman[77755]: 2024-04-14 15:34:09.00141359 +0000 UTC m=+0.364596464 container start be57ad3b61b917f7060108d433c4e1414f77e861892c01c82d010b98bbeb7b96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:34:09.074 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 podman[77755]: 2024-04-14 15:34:09.07331199 +0000 UTC m=+0.436494891 container attach be57ad3b61b917f7060108d433c4e1414f77e861892c01c82d010b98bbeb7b96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-14T15:34:09.819 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate[77770]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T15:34:09.819 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate[77770]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T15:34:09.819 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 bash[77755]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T15:34:09.819 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 bash[77755]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T15:34:09.820 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 bash[77755]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T15:34:09.820 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 bash[77755]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T15:34:09.820 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 bash[77755]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-14T15:34:09.820 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 bash[77755]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T15:34:09.820 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 bash[77755]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-14T15:34:09.820 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate[77770]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T15:34:09.820 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate[77770]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T15:34:09.820 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate[77770]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-14T15:34:09.820 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate[77770]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T15:34:09.820 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate[77770]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-14T15:34:09.820 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:09 smithi016 podman[77755]: 2024-04-14 15:34:09.475092884 +0000 UTC m=+0.838275757 container died be57ad3b61b917f7060108d433c4e1414f77e861892c01c82d010b98bbeb7b96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-14T15:34:10.456 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:10 smithi016 podman[77897]: 2024-04-14 15:34:10.119546793 +0000 UTC m=+0.639352374 container remove be57ad3b61b917f7060108d433c4e1414f77e861892c01c82d010b98bbeb7b96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-14T15:34:10.457 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:10 smithi016 podman[77963]: 2024-04-14 15:34:10.24150618 +0000 UTC m=+0.018758201 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:34:10.457 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:10 smithi016 podman[77963]: 2024-04-14 15:34:10.353963557 +0000 UTC m=+0.131215554 container create 6925155fa41a535cb5833ea796a0d9984e87b0cb4b80fe20ab03fde8e94c4761 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-14T15:34:10.860 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:10 smithi016 podman[77963]: 2024-04-14 15:34:10.598168908 +0000 UTC m=+0.375420909 container init 6925155fa41a535cb5833ea796a0d9984e87b0cb4b80fe20ab03fde8e94c4761 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-14T15:34:10.861 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:10 smithi016 podman[77963]: 2024-04-14 15:34:10.601589887 +0000 UTC m=+0.378841887 container start 6925155fa41a535cb5833ea796a0d9984e87b0cb4b80fe20ab03fde8e94c4761 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403) 2024-04-14T15:34:10.861 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:10 smithi016 ceph-osd[77983]: -- 172.21.15.16:0/3821560553 <== mon.1 v2:172.21.15.16:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c782653680 con 0x55c78261d000 2024-04-14T15:34:10.861 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:10 smithi016 bash[77963]: 6925155fa41a535cb5833ea796a0d9984e87b0cb4b80fe20ab03fde8e94c4761 2024-04-14T15:34:10.861 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:10 smithi016 systemd[1]: Started Ceph osd.3 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:34:11.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:10 smithi016 ceph-mon[64807]: pgmap v103: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:11.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:10 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:11.139 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:10 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:11.140 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:10 smithi016 ceph-mon[67348]: pgmap v103: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:11.140 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:10 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:11.140 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:10 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:11.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:10 smithi196 ceph-mon[58679]: pgmap v103: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:11.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:10 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:11.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:10 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:11.567 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[77978]: 2024-04-14T15:34:11.138+0000 7f763ee797c0 -1 Falling back to public interface 2024-04-14T15:34:11.834 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:34:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:34:11] "GET /metrics HTTP/1.1" 200 38293 "" "Prometheus/2.33.4" 2024-04-14T15:34:13.036 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:34:13.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:12 smithi196 ceph-mon[58679]: pgmap v104: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:13.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:12 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:13.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:12 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:13.358 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:12 smithi016 ceph-mon[67348]: pgmap v104: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:13.358 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:13.358 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:13.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:12 smithi016 ceph-mon[64807]: pgmap v104: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:13.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:13.359 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:13.749 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (4m) 18s ago 9m 23.9M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 unknown 108s ago 8m - - 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (3m) 18s ago 8m 52.2M - 3.5 e1d6a67b021e 27d95fa4403f 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (2m) 108s ago 13m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (3m) 18s ago 15m 507M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (2m) 18s ago 15m 47.2M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (119s) 108s ago 13m 22.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 72817f80f666 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (2m) 18s ago 13m 39.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (4m) 18s ago 9m 19.5M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (3m) 108s ago 9m 18.0M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (64s) 18s ago 13m 42.6M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 f6ee7262903f 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (43s) 18s ago 12m 43.2M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 14dec0c76aaf 2024-04-14T15:34:13.750 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (23s) 18s ago 12m 15.0M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 ce205c8399d3 2024-04-14T15:34:13.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 starting - - - 2737M 2024-04-14T15:34:13.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (11m) 108s ago 11m 54.4M 2992M 17.2.0 e1d6a67b021e 2efd9dba419c 2024-04-14T15:34:13.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (11m) 108s ago 11m 50.8M 2992M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:34:13.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (10m) 108s ago 10m 49.6M 2992M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:34:13.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (10m) 108s ago 10m 54.7M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:34:13.751 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (3m) 108s ago 9m 71.0M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:34:13.751 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (8m) 18s ago 8m 75.2M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:34:13.751 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (8m) 108s ago 8m 74.9M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:34:13.751 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (8m) 18s ago 8m 73.1M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:34:13.752 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (8m) 108s ago 8m 75.3M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:34:13.886 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:14.111 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:13 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[77978]: 2024-04-14T15:34:13.869+0000 7f763ee797c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-14T15:34:14.111 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:13 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[77978]: 2024-04-14T15:34:13.915+0000 7f763ee797c0 -1 osd.3 120 log_to_monitors true 2024-04-14T15:34:14.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:13 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:14.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:13 smithi016 ceph-mon[67348]: from='client.54268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:14.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:13 smithi016 ceph-mon[67348]: from='client.44307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:14.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:13 smithi016 ceph-mon[67348]: from='client.44313 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:14.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:13 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:14.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:13 smithi016 ceph-mon[64807]: from='client.54268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:14.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:13 smithi016 ceph-mon[64807]: from='client.44307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:14.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:13 smithi016 ceph-mon[64807]: from='client.44313 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:14.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:13 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:14.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:13 smithi196 ceph-mon[58679]: from='client.54268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:14.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:13 smithi196 ceph-mon[58679]: from='client.44307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:14.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:13 smithi196 ceph-mon[58679]: from='client.44313 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:14.206 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:34:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:34:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:34:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:34:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:34:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:34:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:34:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:34:14.206 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-14T15:34:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:34:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:34:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:34:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:34:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:34:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:34:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-14T15:34:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T15:34:14.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:34:14.207 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:34:14.554 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:34:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:34:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:34:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:34:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:34:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-04-14T15:34:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-14T15:34:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:34:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "8/23 daemons upgraded", 2024-04-14T15:34:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T15:34:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:34:14.555 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:34:14.992 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded 2024-04-14T15:34:14.992 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:34:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in unknown state 2024-04-14T15:34:14.992 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-14T15:34:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: osd.3 (root=default,host=smithi016) is down 2024-04-14T15:34:14.992 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded 2024-04-14T15:34:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-04-14T15:34:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.a is active+undersized+degraded, acting [1,7] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.e is active+undersized+degraded, acting [2,7] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.14 is active+undersized+degraded, acting [6,5] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-04-14T15:34:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-04-14T15:34:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-04-14T15:34:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-04-14T15:34:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-04-14T15:34:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-04-14T15:34:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-04-14T15:34:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: pg 4.9 is active+undersized+degraded, acting [4,1] 2024-04-14T15:34:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: pg 4.14 is active+undersized+degraded, acting [1,7] 2024-04-14T15:34:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: pg 4.15 is active+undersized+degraded, acting [5,7] 2024-04-14T15:34:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: pg 5.1c is active+undersized+degraded, acting [4,2] 2024-04-14T15:34:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: pg 6.c is active+undersized+degraded, acting [6,5] 2024-04-14T15:34:15.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:14 smithi016 ceph-mon[67348]: pgmap v105: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:15.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:14 smithi016 ceph-mon[67348]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:34:15.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:14 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/3428511825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:15.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:14 smithi016 ceph-mon[67348]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:15.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:14 smithi016 ceph-mon[64807]: pgmap v105: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:15.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:14 smithi016 ceph-mon[64807]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:34:15.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:14 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/3428511825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:15.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:14 smithi016 ceph-mon[64807]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:15.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:14 smithi196 ceph-mon[58679]: pgmap v105: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:15.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:14 smithi196 ceph-mon[58679]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T15:34:15.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:14 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/3428511825' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:15.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:14 smithi196 ceph-mon[58679]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:15.610 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[77978]: 2024-04-14T15:34:15.158+0000 7f762c2c9700 -1 osd.3 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:34:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:15 smithi196 ceph-mon[58679]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T15:34:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:15 smithi196 ceph-mon[58679]: osdmap e123: 8 total, 7 up, 8 in 2024-04-14T15:34:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:15 smithi196 ceph-mon[58679]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:34:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:15 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/3832758262' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:34:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:15 smithi196 ceph-mon[58679]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' 2024-04-14T15:34:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:15 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:16.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:15 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:16.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[67348]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T15:34:16.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[67348]: osdmap e123: 8 total, 7 up, 8 in 2024-04-14T15:34:16.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[67348]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:34:16.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/3832758262' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:34:16.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[67348]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' 2024-04-14T15:34:16.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:16.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:16.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[64807]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T15:34:16.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[64807]: osdmap e123: 8 total, 7 up, 8 in 2024-04-14T15:34:16.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[64807]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-04-14T15:34:16.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/3832758262' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:34:16.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[64807]: from='osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560]' entity='osd.3' 2024-04-14T15:34:16.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:16.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:15 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:17.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:16 smithi196 ceph-mon[58679]: pgmap v107: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:17.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:16 smithi196 ceph-mon[58679]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:34:17.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:16 smithi196 ceph-mon[58679]: osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560] boot 2024-04-14T15:34:17.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:16 smithi196 ceph-mon[58679]: osdmap e124: 8 total, 8 up, 8 in 2024-04-14T15:34:17.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:34:17.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:17.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:17.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:16 smithi196 ceph-mon[58679]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:17.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[67348]: pgmap v107: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:17.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[67348]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:34:17.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[67348]: osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560] boot 2024-04-14T15:34:17.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[67348]: osdmap e124: 8 total, 8 up, 8 in 2024-04-14T15:34:17.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:34:17.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:17.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:17.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[67348]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:17.170 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[64807]: pgmap v107: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T15:34:17.170 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[64807]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:34:17.170 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[64807]: osd.3 [v2:172.21.15.16:6826/83351560,v1:172.21.15.16:6827/83351560] boot 2024-04-14T15:34:17.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[64807]: osdmap e124: 8 total, 8 up, 8 in 2024-04-14T15:34:17.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T15:34:17.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:17.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:17.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:16 smithi016 ceph-mon[64807]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:18.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:18 smithi196 ceph-mon[58679]: osdmap e125: 8 total, 8 up, 8 in 2024-04-14T15:34:18.449 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:18 smithi016 ceph-mon[67348]: osdmap e125: 8 total, 8 up, 8 in 2024-04-14T15:34:18.449 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:18 smithi016 ceph-mon[64807]: osdmap e125: 8 total, 8 up, 8 in 2024-04-14T15:34:18.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:19.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:19 smithi196 ceph-mon[58679]: pgmap v110: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/627 objects degraded (9.410%) 2024-04-14T15:34:19.567 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:19 smithi016 ceph-mon[67348]: pgmap v110: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/627 objects degraded (9.410%) 2024-04-14T15:34:19.567 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:19 smithi016 ceph-mon[64807]: pgmap v110: 161 pgs: 17 active+undersized+degraded, 30 active+undersized, 114 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/627 objects degraded (9.410%) 2024-04-14T15:34:21.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: pgmap v111: 161 pgs: 10 active+undersized+degraded, 17 active+undersized, 134 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 35/627 objects degraded (5.582%) 2024-04-14T15:34:21.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:21.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:21.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:21.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:34:21.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:21.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:21.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:21.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:34:21.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:21.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:21.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:21.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T15:34:21.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T15:34:21.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: Upgrade: osd.4 is safe to restart 2024-04-14T15:34:21.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[67348]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded) 2024-04-14T15:34:21.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: pgmap v111: 161 pgs: 10 active+undersized+degraded, 17 active+undersized, 134 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 35/627 objects degraded (5.582%) 2024-04-14T15:34:21.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:21.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:21.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: Upgrade: osd.4 is safe to restart 2024-04-14T15:34:21.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:20 smithi016 ceph-mon[64807]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded) 2024-04-14T15:34:21.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: pgmap v111: 161 pgs: 10 active+undersized+degraded, 17 active+undersized, 134 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 35/627 objects degraded (5.582%) 2024-04-14T15:34:21.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:21.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: Upgrade: osd.4 is safe to restart 2024-04-14T15:34:21.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:20 smithi196 ceph-mon[58679]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/627 objects degraded (9.410%), 17 pgs degraded) 2024-04-14T15:34:21.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:34:21 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:34:21] "GET /metrics HTTP/1.1" 200 38444 "" "Prometheus/2.33.4" 2024-04-14T15:34:23.011 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:22 smithi196 ceph-mon[58679]: Upgrade: Updating osd.4 2024-04-14T15:34:23.011 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:22 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:23.011 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:22 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:34:23.011 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:22 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:23.011 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:22 smithi196 ceph-mon[58679]: Deploying daemon osd.4 on smithi196 2024-04-14T15:34:23.011 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:22 smithi196 ceph-mon[58679]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:23.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[64807]: Upgrade: Updating osd.4 2024-04-14T15:34:23.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:23.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:34:23.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:23.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[64807]: Deploying daemon osd.4 on smithi196 2024-04-14T15:34:23.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[64807]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:23.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[67348]: Upgrade: Updating osd.4 2024-04-14T15:34:23.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:23.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T15:34:23.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:23.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[67348]: Deploying daemon osd.4 on smithi196 2024-04-14T15:34:23.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:22 smithi016 ceph-mon[67348]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:23.631 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:23 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:23.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:23 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:24.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:23 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:24.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:23 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:24.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:24 smithi016 ceph-mon[67348]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:24.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:24 smithi016 ceph-mon[64807]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:24.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:24 smithi196 ceph-mon[58679]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:34:25.223 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:24 smithi196 systemd[1]: Stopping Ceph osd.4 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:34:25.665 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:25 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[29135]: 2024-04-14T15:34:25.222+0000 7ffba6fd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:34:25.665 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:25 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[29135]: 2024-04-14T15:34:25.222+0000 7ffba6fd8700 -1 osd.4 125 *** Got signal Terminated *** 2024-04-14T15:34:25.665 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:25 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[29135]: 2024-04-14T15:34:25.222+0000 7ffba6fd8700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:34:26.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:25 smithi016 ceph-mon[67348]: osd.4 marked itself down and dead 2024-04-14T15:34:26.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:25 smithi016 ceph-mon[64807]: osd.4 marked itself down and dead 2024-04-14T15:34:26.111 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:25 smithi196 ceph-mon[58679]: osd.4 marked itself down and dead 2024-04-14T15:34:26.374 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:26 smithi196 podman[61869]: 2024-04-14 15:34:26.111472815 +0000 UTC m=+0.995301534 container died 2efd9dba419c9e6ad5093ea51c6d729833299d805e83bd28885653aeb461d573 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_CLEAN=True, RELEASE=HEAD, version=8, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.component=centos-stream-container, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, name=centos-stream, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream) 2024-04-14T15:34:26.665 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:26 smithi196 podman[61869]: 2024-04-14 15:34:26.556484 +0000 UTC m=+1.440312749 container cleanup 2efd9dba419c9e6ad5093ea51c6d729833299d805e83bd28885653aeb461d573 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_CLEAN=True, io.buildah.version=1.19.8, version=8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, vcs-type=git, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., RELEASE=HEAD) 2024-04-14T15:34:26.666 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:26 smithi196 bash[61869]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4 2024-04-14T15:34:27.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:26 smithi016 ceph-mon[67348]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:34:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:26 smithi016 ceph-mon[67348]: osdmap e126: 8 total, 7 up, 8 in 2024-04-14T15:34:27.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:26 smithi016 ceph-mon[67348]: pgmap v117: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-04-14T15:34:27.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:26 smithi016 ceph-mon[64807]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:34:27.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:26 smithi016 ceph-mon[64807]: osdmap e126: 8 total, 7 up, 8 in 2024-04-14T15:34:27.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:26 smithi016 ceph-mon[64807]: pgmap v117: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-04-14T15:34:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:26 smithi196 ceph-mon[58679]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:34:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:26 smithi196 ceph-mon[58679]: osdmap e126: 8 total, 7 up, 8 in 2024-04-14T15:34:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:26 smithi196 ceph-mon[58679]: pgmap v117: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-04-14T15:34:27.165 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:26 smithi196 podman[61887]: 2024-04-14 15:34:26.774555369 +0000 UTC m=+0.659393554 container remove 2efd9dba419c9e6ad5093ea51c6d729833299d805e83bd28885653aeb461d573 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., version=8, ceph=True, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.openshift.expose-services=) 2024-04-14T15:34:27.511 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:27 smithi196 podman[61980]: 2024-04-14 15:34:27.090612644 +0000 UTC m=+0.017587184 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:34:27.511 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:27 smithi196 podman[61980]: 2024-04-14 15:34:27.217041502 +0000 UTC m=+0.144016020 container create b3622b9d4b03aac91625c97307eb904661a7a91798bacc00944dec16fb681816 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-14T15:34:27.511 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:27 smithi196 podman[61980]: 2024-04-14 15:34:27.419487577 +0000 UTC m=+0.346462096 container init b3622b9d4b03aac91625c97307eb904661a7a91798bacc00944dec16fb681816 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-14T15:34:27.511 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:27 smithi196 podman[61980]: 2024-04-14 15:34:27.422539345 +0000 UTC m=+0.349513873 container start b3622b9d4b03aac91625c97307eb904661a7a91798bacc00944dec16fb681816 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:34:27.511 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:27 smithi196 podman[61980]: 2024-04-14 15:34:27.511064003 +0000 UTC m=+0.438038536 container attach b3622b9d4b03aac91625c97307eb904661a7a91798bacc00944dec16fb681816 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403) 2024-04-14T15:34:27.789 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:27 smithi196 podman[61980]: 2024-04-14 15:34:27.712634813 +0000 UTC m=+0.639609326 container died b3622b9d4b03aac91625c97307eb904661a7a91798bacc00944dec16fb681816 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-14T15:34:28.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:27 smithi016 ceph-mon[67348]: osdmap e127: 8 total, 7 up, 8 in 2024-04-14T15:34:28.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:28.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:28.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:28.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:27 smithi016 ceph-mon[64807]: osdmap e127: 8 total, 7 up, 8 in 2024-04-14T15:34:28.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:28.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:28.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:27 smithi196 ceph-mon[58679]: osdmap e127: 8 total, 7 up, 8 in 2024-04-14T15:34:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:28.580 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:28 smithi196 podman[62005]: 2024-04-14 15:34:28.365482666 +0000 UTC m=+0.643739150 container remove b3622b9d4b03aac91625c97307eb904661a7a91798bacc00944dec16fb681816 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=) 2024-04-14T15:34:28.580 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:28 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-14T15:34:28.580 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:28 smithi196 systemd[1]: Stopped Ceph osd.4 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:34:28.580 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:28 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.4.service: Consumed 8.366s CPU time. 2024-04-14T15:34:28.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:28 smithi196 ceph-mon[58679]: pgmap v119: 161 pgs: 14 active+undersized+degraded, 14 stale+active+clean, 18 active+undersized, 115 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-04-14T15:34:28.916 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:28 smithi196 systemd[1]: Starting Ceph osd.4 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:34:28.916 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:28 smithi196 podman[62111]: 2024-04-14 15:34:28.700629902 +0000 UTC m=+0.018250705 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:34:28.916 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:28 smithi196 podman[62111]: 2024-04-14 15:34:28.816870432 +0000 UTC m=+0.134491273 container create d8fb1887dbb267b6a83023f89206bd5391bf3c3afdb518decc834c97ba1419d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate, ceph=True, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-14T15:34:28.916 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:28 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:28.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-14T15:34:28.916 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:28 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:29.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:28 smithi016 ceph-mon[67348]: pgmap v119: 161 pgs: 14 active+undersized+degraded, 14 stale+active+clean, 18 active+undersized, 115 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-04-14T15:34:29.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:28 smithi016 ceph-mon[64807]: pgmap v119: 161 pgs: 14 active+undersized+degraded, 14 stale+active+clean, 18 active+undersized, 115 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-04-14T15:34:29.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 podman[62111]: 2024-04-14 15:34:29.069381428 +0000 UTC m=+0.387002224 container init d8fb1887dbb267b6a83023f89206bd5391bf3c3afdb518decc834c97ba1419d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:34:29.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 podman[62111]: 2024-04-14 15:34:29.072407098 +0000 UTC m=+0.390027894 container start d8fb1887dbb267b6a83023f89206bd5391bf3c3afdb518decc834c97ba1419d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:34:29.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 podman[62111]: 2024-04-14 15:34:29.169196765 +0000 UTC m=+0.486817583 container attach d8fb1887dbb267b6a83023f89206bd5391bf3c3afdb518decc834c97ba1419d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:34:29.796 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate[62127]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T15:34:29.796 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate[62127]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T15:34:29.796 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 bash[62111]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T15:34:29.796 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 bash[62111]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T15:34:29.796 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 bash[62111]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T15:34:29.796 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 bash[62111]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T15:34:29.796 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 bash[62111]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-14T15:34:29.796 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 bash[62111]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T15:34:29.797 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 bash[62111]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-14T15:34:29.797 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate[62127]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T15:34:29.797 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate[62127]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T15:34:29.797 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate[62127]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-14T15:34:29.797 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate[62127]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T15:34:29.797 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate[62127]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-14T15:34:29.797 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:29 smithi196 podman[62111]: 2024-04-14 15:34:29.537436061 +0000 UTC m=+0.855056857 container died d8fb1887dbb267b6a83023f89206bd5391bf3c3afdb518decc834c97ba1419d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:34:30.089 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:29 smithi196 ceph-mon[58679]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 14 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:30.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:29 smithi016 ceph-mon[67348]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 14 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:30.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:29 smithi016 ceph-mon[64807]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 14 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:30.340 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:30 smithi196 podman[62254]: 2024-04-14 15:34:30.089725422 +0000 UTC m=+0.544746250 container remove d8fb1887dbb267b6a83023f89206bd5391bf3c3afdb518decc834c97ba1419d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-activate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, ceph=True) 2024-04-14T15:34:30.340 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:30 smithi196 podman[62320]: 2024-04-14 15:34:30.205271103 +0000 UTC m=+0.018329418 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:34:30.599 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:30 smithi196 podman[62320]: 2024-04-14 15:34:30.34060139 +0000 UTC m=+0.153659692 container create 3a7864fa8601b80572c94dd3220e0b12a5afc98d68fe90e68488dbc5eae15331 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-14T15:34:30.599 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:30 smithi196 podman[62320]: 2024-04-14 15:34:30.57629037 +0000 UTC m=+0.389348669 container init 3a7864fa8601b80572c94dd3220e0b12a5afc98d68fe90e68488dbc5eae15331 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:34:30.599 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:30 smithi196 podman[62320]: 2024-04-14 15:34:30.579111089 +0000 UTC m=+0.392169389 container start 3a7864fa8601b80572c94dd3220e0b12a5afc98d68fe90e68488dbc5eae15331 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, io.buildah.version=1.33.5) 2024-04-14T15:34:30.915 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:30 smithi196 ceph-osd[62339]: -- 172.21.15.196:0/3684025927 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55921e3a5680 con 0x55921e36ec00 2024-04-14T15:34:30.915 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:30 smithi196 bash[62320]: 3a7864fa8601b80572c94dd3220e0b12a5afc98d68fe90e68488dbc5eae15331 2024-04-14T15:34:30.915 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:30 smithi196 systemd[1]: Started Ceph osd.4 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:34:31.414 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:31 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:34:31.123+0000 7f3f398827c0 -1 Falling back to public interface 2024-04-14T15:34:31.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:31 smithi016 ceph-mon[67348]: pgmap v120: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:31.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:31 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:31.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:31 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:31.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:31 smithi016 ceph-mon[64807]: pgmap v120: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:31.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:31 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:31.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:31 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:31.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:34:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:34:31] "GET /metrics HTTP/1.1" 200 38443 "" "Prometheus/2.33.4" 2024-04-14T15:34:31.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:31 smithi196 ceph-mon[58679]: pgmap v120: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:31.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:31 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:31.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:31 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:33.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:33 smithi196 ceph-mon[58679]: pgmap v121: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:33.812 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:33.506Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:34.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:33 smithi016 ceph-mon[67348]: pgmap v121: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:34.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:33 smithi016 ceph-mon[64807]: pgmap v121: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:34.164 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:34:33.810+0000 7f3f398827c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-14T15:34:34.164 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:34:33.847+0000 7f3f398827c0 -1 osd.4 125 log_to_monitors true 2024-04-14T15:34:35.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:34 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:35.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:34 smithi016 ceph-mon[67348]: from='osd.4 [v2:172.21.15.196:6800/634358266,v1:172.21.15.196:6801/634358266]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:34:35.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:34 smithi016 ceph-mon[67348]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:34:35.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:34 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:34 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:34 smithi016 ceph-mon[64807]: from='osd.4 [v2:172.21.15.196:6800/634358266,v1:172.21.15.196:6801/634358266]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:34:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:34 smithi016 ceph-mon[64807]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:34:35.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:34 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:35.165 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:34:34 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:34:34.784+0000 7f3f2acda700 -1 osd.4 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:34:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:34 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:34 smithi196 ceph-mon[58679]: from='osd.4 [v2:172.21.15.196:6800/634358266,v1:172.21.15.196:6801/634358266]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:34:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:34 smithi196 ceph-mon[58679]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T15:34:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:34 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[64807]: pgmap v122: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[64807]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T15:34:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[64807]: osdmap e128: 8 total, 7 up, 8 in 2024-04-14T15:34:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[64807]: from='osd.4 [v2:172.21.15.196:6800/634358266,v1:172.21.15.196:6801/634358266]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:34:36.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[64807]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:34:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[67348]: pgmap v122: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[67348]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T15:34:36.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[67348]: osdmap e128: 8 total, 7 up, 8 in 2024-04-14T15:34:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[67348]: from='osd.4 [v2:172.21.15.196:6800/634358266,v1:172.21.15.196:6801/634358266]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:34:36.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:35 smithi016 ceph-mon[67348]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:34:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:35 smithi196 ceph-mon[58679]: pgmap v122: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:35 smithi196 ceph-mon[58679]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T15:34:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:35 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:35 smithi196 ceph-mon[58679]: osdmap e128: 8 total, 7 up, 8 in 2024-04-14T15:34:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:35 smithi196 ceph-mon[58679]: from='osd.4 [v2:172.21.15.196:6800/634358266,v1:172.21.15.196:6801/634358266]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:34:36.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:35 smithi196 ceph-mon[58679]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:34:37.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[67348]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:34:37.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[67348]: osd.4 [v2:172.21.15.196:6800/634358266,v1:172.21.15.196:6801/634358266] boot 2024-04-14T15:34:37.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[67348]: osdmap e129: 8 total, 8 up, 8 in 2024-04-14T15:34:37.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:34:37.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:37.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:37.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[67348]: pgmap v125: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:37.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[67348]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 27 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:37.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[64807]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:34:37.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[64807]: osd.4 [v2:172.21.15.196:6800/634358266,v1:172.21.15.196:6801/634358266] boot 2024-04-14T15:34:37.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[64807]: osdmap e129: 8 total, 8 up, 8 in 2024-04-14T15:34:37.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:34:37.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:37.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:37.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[64807]: pgmap v125: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:37.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:36 smithi016 ceph-mon[64807]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 27 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:36 smithi196 ceph-mon[58679]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:34:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:36 smithi196 ceph-mon[58679]: osd.4 [v2:172.21.15.196:6800/634358266,v1:172.21.15.196:6801/634358266] boot 2024-04-14T15:34:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:36 smithi196 ceph-mon[58679]: osdmap e129: 8 total, 8 up, 8 in 2024-04-14T15:34:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:36 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T15:34:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:36 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:36 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:36 smithi196 ceph-mon[58679]: pgmap v125: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T15:34:37.139 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:36 smithi196 ceph-mon[58679]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 27 pgs degraded (PG_DEGRADED) 2024-04-14T15:34:38.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:37 smithi196 ceph-mon[58679]: osdmap e130: 8 total, 8 up, 8 in 2024-04-14T15:34:38.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:37 smithi016 ceph-mon[64807]: osdmap e130: 8 total, 8 up, 8 in 2024-04-14T15:34:38.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:37 smithi016 ceph-mon[67348]: osdmap e130: 8 total, 8 up, 8 in 2024-04-14T15:34:38.888 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:38 smithi196 ceph-mon[58679]: pgmap v127: 161 pgs: 17 active+undersized+degraded, 28 active+undersized, 116 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-04-14T15:34:38.888 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-14T15:34:38.888 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:39.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:38 smithi016 ceph-mon[67348]: pgmap v127: 161 pgs: 17 active+undersized+degraded, 28 active+undersized, 116 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-04-14T15:34:39.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:38 smithi016 ceph-mon[64807]: pgmap v127: 161 pgs: 17 active+undersized+degraded, 28 active+undersized, 116 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-04-14T15:34:39.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:34:39 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:34:39.431+0000 7fda5603a700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-14T15:34:40.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:40.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:40.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:40.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:34:40.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: pgmap v128: 161 pgs: 14 active+undersized+degraded, 24 active+undersized, 123 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 op/s; 41/627 objects degraded (6.539%) 2024-04-14T15:34:40.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: pgmap v129: 161 pgs: 14 active+undersized+degraded, 24 active+undersized, 123 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41/627 objects degraded (6.539%) 2024-04-14T15:34:40.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:40.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:34:40.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[64807]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: pgmap v128: 161 pgs: 14 active+undersized+degraded, 24 active+undersized, 123 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 op/s; 41/627 objects degraded (6.539%) 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: pgmap v129: 161 pgs: 14 active+undersized+degraded, 24 active+undersized, 123 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41/627 objects degraded (6.539%) 2024-04-14T15:34:40.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:40.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:34:40.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:40.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:40.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:40.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:40.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:40.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:40 smithi016 ceph-mon[67348]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-14T15:34:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:34:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: pgmap v128: 161 pgs: 14 active+undersized+degraded, 24 active+undersized, 123 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 op/s; 41/627 objects degraded (6.539%) 2024-04-14T15:34:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: pgmap v129: 161 pgs: 14 active+undersized+degraded, 24 active+undersized, 123 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41/627 objects degraded (6.539%) 2024-04-14T15:34:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:40.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:34:40.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:40.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:40.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:40.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:40.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:40.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:40 smithi196 ceph-mon[58679]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-14T15:34:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:41 smithi196 ceph-mon[58679]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/627 objects degraded (6.539%), 14 pgs degraded) 2024-04-14T15:34:41.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:41 smithi016 ceph-mon[64807]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/627 objects degraded (6.539%), 14 pgs degraded) 2024-04-14T15:34:41.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:41 smithi016 ceph-mon[67348]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/627 objects degraded (6.539%), 14 pgs degraded) 2024-04-14T15:34:41.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:34:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:34:41] "GET /metrics HTTP/1.1" 200 38443 "" "Prometheus/2.33.4" 2024-04-14T15:34:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:42 smithi196 ceph-mon[58679]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-04-14T15:34:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:42.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:42.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:42 smithi016 ceph-mon[64807]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-04-14T15:34:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:42 smithi016 ceph-mon[67348]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-04-14T15:34:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:43.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:44.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:44 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:44.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:44 smithi196 ceph-mon[58679]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-04-14T15:34:44.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:44 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:44.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:44 smithi016 ceph-mon[64807]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-04-14T15:34:44.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:44 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:44.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:44 smithi016 ceph-mon[67348]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-04-14T15:34:45.370 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:34:46.050 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:34:46.050 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (4m) 30s ago 9m 23.9M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:34:46.050 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 error 11s ago 9m - - 2024-04-14T15:34:46.050 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (3m) 30s ago 8m 52.3M - 3.5 e1d6a67b021e 27d95fa4403f 2024-04-14T15:34:46.050 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (3m) 11s ago 14m 451M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:34:46.050 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (4m) 30s ago 15m 508M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (3m) 30s ago 15m 48.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (2m) 11s ago 14m 37.1M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 72817f80f666 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (2m) 30s ago 14m 39.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (4m) 30s ago 10m 17.4M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (4m) 11s ago 10m 17.6M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (96s) 30s ago 13m 44.0M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 f6ee7262903f 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (76s) 30s ago 13m 43.7M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 14dec0c76aaf 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (55s) 30s ago 12m 51.5M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 ce205c8399d3 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (35s) 30s ago 12m 12.5M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 6925155fa41a 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (15s) 11s ago 12m 12.6M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 3a7864fa8601 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (11m) 11s ago 11m 55.4M 2992M 17.2.0 e1d6a67b021e 6ea67735d735 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (11m) 11s ago 11m 54.7M 2992M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:34:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (11m) 11s ago 11m 59.3M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:34:46.052 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (3m) 11s ago 10m 73.4M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:34:46.052 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (8m) 30s ago 8m 75.3M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:34:46.052 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (8m) 11s ago 8m 75.4M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:34:46.052 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (8m) 30s ago 8m 73.2M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:34:46.052 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (8m) 11s ago 8m 75.8M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:34:46.460 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 5 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:34:46.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:34:46.462 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:34:46.462 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-14T15:34:46.462 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 10 2024-04-14T15:34:46.462 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:34:46.462 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:34:46.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:46 smithi196 ceph-mon[58679]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:46.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:46 smithi196 ceph-mon[58679]: from='client.54307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:46.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:46 smithi196 ceph-mon[58679]: from='client.54313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:46.809 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:34:46.809 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:34:46.809 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:34:46.809 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:34:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:34:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-04-14T15:34:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-14T15:34:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:34:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "10/23 daemons upgraded", 2024-04-14T15:34:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T15:34:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:34:46.810 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:34:46.810 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:46 smithi016 ceph-mon[67348]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:46.811 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:46 smithi016 ceph-mon[67348]: from='client.54307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:46.811 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:46 smithi016 ceph-mon[67348]: from='client.54313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:46.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:46 smithi016 ceph-mon[64807]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:46.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:46 smithi016 ceph-mon[64807]: from='client.54307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:46.811 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:46 smithi016 ceph-mon[64807]: from='client.54313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:47.222 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:34:47.222 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:34:47.222 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in error state 2024-04-14T15:34:47.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:47 smithi016 ceph-mon[67348]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:47.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:47 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/2883863725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:47.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:47 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:47.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:47 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/2079165513' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:34:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:47 smithi016 ceph-mon[64807]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:47 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/2883863725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:47 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:47.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:47 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/2079165513' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:34:47.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:47 smithi196 ceph-mon[58679]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:47.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:47 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/2883863725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:34:47.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:47 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:47.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:47 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/2079165513' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:34:48.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:48 smithi196 ceph-mon[58679]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:48.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:48 smithi196 ceph-mon[58679]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:34:48.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:48 smithi016 ceph-mon[67348]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:48.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:48 smithi016 ceph-mon[67348]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:34:48.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:48 smithi016 ceph-mon[64807]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:34:48.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:48 smithi016 ceph-mon[64807]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:34:48.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:48.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-14T15:34:48.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:48.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:50.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:50 smithi196 ceph-mon[58679]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:34:50.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:50 smithi016 ceph-mon[64807]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:34:50.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:50 smithi016 ceph-mon[67348]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:34:51.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:34:51 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:34:51] "GET /metrics HTTP/1.1" 200 38451 "" "Prometheus/2.33.4" 2024-04-14T15:34:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:52 smithi196 ceph-mon[58679]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:52.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:52 smithi016 ceph-mon[64807]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:52.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:52 smithi016 ceph-mon[67348]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:53.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:53 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:54 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:54.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:54 smithi196 ceph-mon[58679]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:34:54.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:54 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:54.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:54 smithi016 ceph-mon[64807]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:34:54.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:54 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:34:54.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:54 smithi016 ceph-mon[67348]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:34:55.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:55 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:55.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:55 smithi196 ceph-mon[58679]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:55.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:55 smithi196 ceph-mon[58679]: Upgrade: osd.5 is safe to restart 2024-04-14T15:34:55.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:55 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:55.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:55 smithi016 ceph-mon[64807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:55.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:55 smithi016 ceph-mon[64807]: Upgrade: osd.5 is safe to restart 2024-04-14T15:34:55.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:55 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:55.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:55 smithi016 ceph-mon[67348]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T15:34:55.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:55 smithi016 ceph-mon[67348]: Upgrade: osd.5 is safe to restart 2024-04-14T15:34:56.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:56 smithi196 ceph-mon[58679]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:56.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:56 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:56.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:56 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:34:56.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:56 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:56.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:56 smithi016 ceph-mon[67348]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:56.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:56 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:56.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:56 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:34:56.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:56 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:56.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:56 smithi016 ceph-mon[64807]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:34:56.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:56 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:34:56.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:56 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T15:34:56.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:56 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:34:57.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:57 smithi196 ceph-mon[58679]: Upgrade: Updating osd.5 2024-04-14T15:34:57.633 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:57 smithi196 ceph-mon[58679]: Deploying daemon osd.5 on smithi196 2024-04-14T15:34:57.634 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:57.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:57 smithi016 ceph-mon[67348]: Upgrade: Updating osd.5 2024-04-14T15:34:57.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:57 smithi016 ceph-mon[67348]: Deploying daemon osd.5 on smithi196 2024-04-14T15:34:57.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:57.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:57 smithi016 ceph-mon[64807]: Upgrade: Updating osd.5 2024-04-14T15:34:57.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:57 smithi016 ceph-mon[64807]: Deploying daemon osd.5 on smithi196 2024-04-14T15:34:57.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:34:58.594 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:34:58 smithi196 ceph-mon[58679]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:34:58.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:34:58 smithi016 ceph-mon[67348]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:34:58.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:34:58 smithi016 ceph-mon[64807]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:34:58.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:34:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:34:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:34:59.580 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:34:59 smithi196 systemd[1]: Stopping Ceph osd.5 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:34:59.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:34:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[32019]: 2024-04-14T15:34:59.579+0000 7f755d974700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:34:59.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:34:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[32019]: 2024-04-14T15:34:59.579+0000 7f755d974700 -1 osd.5 130 *** Got signal Terminated *** 2024-04-14T15:34:59.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:34:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[32019]: 2024-04-14T15:34:59.579+0000 7f755d974700 -1 osd.5 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:35:00.827 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:00 smithi196 ceph-mon[58679]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:00.827 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:00 smithi196 ceph-mon[58679]: osd.5 marked itself down and dead 2024-04-14T15:35:00.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:00 smithi016 ceph-mon[67348]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:00.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:00 smithi016 ceph-mon[67348]: osd.5 marked itself down and dead 2024-04-14T15:35:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:00 smithi016 ceph-mon[64807]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:00 smithi016 ceph-mon[64807]: osd.5 marked itself down and dead 2024-04-14T15:35:01.138 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:00 smithi196 podman[64151]: 2024-04-14 15:35:00.826958237 +0000 UTC m=+1.355919629 container died 6ea67735d7354fe0785237bc422fc888cac15bbc6b2610b5faf7529d22ec9154 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, RELEASE=HEAD, name=centos-stream, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , version=8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, architecture=x86_64, ceph=True, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-14T15:35:01.415 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:01 smithi196 podman[64151]: 2024-04-14 15:35:01.237815284 +0000 UTC m=+1.766776665 container cleanup 6ea67735d7354fe0785237bc422fc888cac15bbc6b2610b5faf7529d22ec9154 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vcs-type=git, release=754, GIT_CLEAN=True, version=8, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-14T15:35:01.415 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:01 smithi196 bash[64151]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5 2024-04-14T15:35:01.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:01 smithi196 ceph-mon[58679]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:35:01.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:01 smithi196 ceph-mon[58679]: osdmap e131: 8 total, 7 up, 8 in 2024-04-14T15:35:01.758 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:01 smithi196 podman[64170]: 2024-04-14 15:35:01.447022359 +0000 UTC m=+0.608237218 container remove 6ea67735d7354fe0785237bc422fc888cac15bbc6b2610b5faf7529d22ec9154 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5, distribution-scope=public, io.buildah.version=1.19.8, release=754, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, architecture=x86_64, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, version=8, RELEASE=HEAD, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-14T15:35:01.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:35:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:35:01] "GET /metrics HTTP/1.1" 200 38452 "" "Prometheus/2.33.4" 2024-04-14T15:35:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:01 smithi016 ceph-mon[67348]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:35:01.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:01 smithi016 ceph-mon[67348]: osdmap e131: 8 total, 7 up, 8 in 2024-04-14T15:35:01.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:01 smithi016 ceph-mon[64807]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:35:01.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:01 smithi016 ceph-mon[64807]: osdmap e131: 8 total, 7 up, 8 in 2024-04-14T15:35:02.050 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:01 smithi196 podman[64257]: 2024-04-14 15:35:01.66065357 +0000 UTC m=+0.017735418 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:35:02.051 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:01 smithi196 podman[64257]: 2024-04-14 15:35:01.764644953 +0000 UTC m=+0.121726791 container create 7e45f90a3f2085bf6bafef926be936a006226c4addb28ff05e9fcc93f0da3e93 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:35:02.051 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:01 smithi196 podman[64257]: 2024-04-14 15:35:01.96733903 +0000 UTC m=+0.324420874 container init 7e45f90a3f2085bf6bafef926be936a006226c4addb28ff05e9fcc93f0da3e93 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:35:02.051 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:01 smithi196 podman[64257]: 2024-04-14 15:35:01.970555626 +0000 UTC m=+0.327637467 container start 7e45f90a3f2085bf6bafef926be936a006226c4addb28ff05e9fcc93f0da3e93 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:35:02.051 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:02 smithi196 podman[64257]: 2024-04-14 15:35:02.050696056 +0000 UTC m=+0.407777900 container attach 7e45f90a3f2085bf6bafef926be936a006226c4addb28ff05e9fcc93f0da3e93 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-14T15:35:02.415 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:02 smithi196 podman[64257]: 2024-04-14 15:35:02.255498238 +0000 UTC m=+0.612580076 container died 7e45f90a3f2085bf6bafef926be936a006226c4addb28ff05e9fcc93f0da3e93 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:35:02.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:02 smithi016 ceph-mon[67348]: pgmap v141: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:35:02.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:02 smithi016 ceph-mon[67348]: osdmap e132: 8 total, 7 up, 8 in 2024-04-14T15:35:02.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:02 smithi016 ceph-mon[64807]: pgmap v141: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:35:02.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:02 smithi016 ceph-mon[64807]: osdmap e132: 8 total, 7 up, 8 in 2024-04-14T15:35:02.906 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:02 smithi196 ceph-mon[58679]: pgmap v141: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:35:02.906 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:02 smithi196 ceph-mon[58679]: osdmap e132: 8 total, 7 up, 8 in 2024-04-14T15:35:03.165 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:02 smithi196 podman[64283]: 2024-04-14 15:35:02.906898014 +0000 UTC m=+0.645505824 container remove 7e45f90a3f2085bf6bafef926be936a006226c4addb28ff05e9fcc93f0da3e93 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-14T15:35:03.165 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:02 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-14T15:35:03.165 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:02 smithi196 systemd[1]: Stopped Ceph osd.5 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:35:03.165 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:02 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.5.service: Consumed 8.623s CPU time. 2024-04-14T15:35:03.165 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:03 smithi196 systemd[1]: Starting Ceph osd.5 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:35:03.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:03 smithi196 ceph-mon[58679]: Health check failed: Degraded data redundancy: 18/627 objects degraded (2.871%), 7 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:03.649 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:35:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:35:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:35:03.649 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:03 smithi196 podman[64385]: 2024-04-14 15:35:03.347126353 +0000 UTC m=+0.114285670 container create 3089d1fc6fdc11f742dd92f36d0b1f04faa676666df38f8d69052810ec0791ff (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, ceph=True, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:35:03.650 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:03 smithi196 podman[64385]: 2024-04-14 15:35:03.250478254 +0000 UTC m=+0.017637576 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:35:03.650 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:03 smithi196 podman[64385]: 2024-04-14 15:35:03.575013716 +0000 UTC m=+0.342173043 container init 3089d1fc6fdc11f742dd92f36d0b1f04faa676666df38f8d69052810ec0791ff (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate, ceph=True, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-14T15:35:03.650 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:03 smithi196 podman[64385]: 2024-04-14 15:35:03.57841113 +0000 UTC m=+0.345570448 container start 3089d1fc6fdc11f742dd92f36d0b1f04faa676666df38f8d69052810ec0791ff (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate, ceph=True, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-14T15:35:03.650 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:03 smithi196 podman[64385]: 2024-04-14 15:35:03.649526743 +0000 UTC m=+0.416686080 container attach 3089d1fc6fdc11f742dd92f36d0b1f04faa676666df38f8d69052810ec0791ff (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-14T15:35:03.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:03 smithi016 ceph-mon[64807]: Health check failed: Degraded data redundancy: 18/627 objects degraded (2.871%), 7 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:03.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:03 smithi016 ceph-mon[67348]: Health check failed: Degraded data redundancy: 18/627 objects degraded (2.871%), 7 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:04.361 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate[64399]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T15:35:04.362 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate[64399]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T15:35:04.362 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 bash[64385]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T15:35:04.362 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 bash[64385]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T15:35:04.362 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 bash[64385]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T15:35:04.362 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 bash[64385]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T15:35:04.362 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 bash[64385]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-14T15:35:04.362 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 bash[64385]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T15:35:04.362 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 bash[64385]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-14T15:35:04.362 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate[64399]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T15:35:04.362 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate[64399]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T15:35:04.362 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate[64399]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-14T15:35:04.363 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate[64399]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T15:35:04.363 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate[64399]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-14T15:35:04.363 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 podman[64385]: 2024-04-14 15:35:04.051101007 +0000 UTC m=+0.818260313 container died 3089d1fc6fdc11f742dd92f36d0b1f04faa676666df38f8d69052810ec0791ff (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-14T15:35:04.664 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:04 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:04.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:04 smithi196 ceph-mon[58679]: pgmap v143: 161 pgs: 7 active+undersized+degraded, 17 stale+active+clean, 9 active+undersized, 128 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 18/627 objects degraded (2.871%) 2024-04-14T15:35:04.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:04 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:04 smithi016 ceph-mon[67348]: pgmap v143: 161 pgs: 7 active+undersized+degraded, 17 stale+active+clean, 9 active+undersized, 128 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 18/627 objects degraded (2.871%) 2024-04-14T15:35:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:04 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:04 smithi016 ceph-mon[64807]: pgmap v143: 161 pgs: 7 active+undersized+degraded, 17 stale+active+clean, 9 active+undersized, 128 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 18/627 objects degraded (2.871%) 2024-04-14T15:35:05.001 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 podman[64532]: 2024-04-14 15:35:04.67031589 +0000 UTC m=+0.613740154 container remove 3089d1fc6fdc11f742dd92f36d0b1f04faa676666df38f8d69052810ec0791ff (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-activate, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5) 2024-04-14T15:35:05.001 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 podman[64600]: 2024-04-14 15:35:04.785288083 +0000 UTC m=+0.018108402 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:35:05.002 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:04 smithi196 podman[64600]: 2024-04-14 15:35:04.896099398 +0000 UTC m=+0.128919707 container create 08ddd21e09299f74a76e7de359297138040d9b5d2c8aa27fc51330220f9c27ed (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2) 2024-04-14T15:35:05.415 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:05 smithi196 podman[64600]: 2024-04-14 15:35:05.090293863 +0000 UTC m=+0.323114173 container init 08ddd21e09299f74a76e7de359297138040d9b5d2c8aa27fc51330220f9c27ed (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-14T15:35:05.415 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:05 smithi196 podman[64600]: 2024-04-14 15:35:05.093222726 +0000 UTC m=+0.326043036 container start 08ddd21e09299f74a76e7de359297138040d9b5d2c8aa27fc51330220f9c27ed (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-14T15:35:05.415 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:05 smithi196 ceph-osd[64619]: -- 172.21.15.196:0/538636994 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c284ff9680 con 0x55c284fc3000 2024-04-14T15:35:05.415 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:05 smithi196 bash[64600]: 08ddd21e09299f74a76e7de359297138040d9b5d2c8aa27fc51330220f9c27ed 2024-04-14T15:35:05.415 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:05 smithi196 systemd[1]: Started Ceph osd.5 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:35:05.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:35:05.640+0000 7f4ca93c97c0 -1 Falling back to public interface 2024-04-14T15:35:06.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:06 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:06 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:06 smithi196 ceph-mon[58679]: pgmap v144: 161 pgs: 22 active+undersized+degraded, 38 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-04-14T15:35:06.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:06 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:06.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:06 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:06.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:06 smithi016 ceph-mon[67348]: pgmap v144: 161 pgs: 22 active+undersized+degraded, 38 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-04-14T15:35:06.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:06 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:06.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:06 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:06.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:06 smithi016 ceph-mon[64807]: pgmap v144: 161 pgs: 22 active+undersized+degraded, 38 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-04-14T15:35:08.659 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:35:08.346+0000 7f4ca93c97c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-14T15:35:08.659 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:35:08.370+0000 7f4ca93c97c0 -1 osd.5 130 log_to_monitors true 2024-04-14T15:35:08.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:08 smithi196 ceph-mon[58679]: pgmap v145: 161 pgs: 22 active+undersized+degraded, 38 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-04-14T15:35:08.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:08 smithi196 ceph-mon[58679]: from='osd.5 [v2:172.21.15.196:6808/2286579913,v1:172.21.15.196:6809/2286579913]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:35:08.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:08 smithi196 ceph-mon[58679]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:35:08.916 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:35:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:35:08.697+0000 7f4c9a821700 -1 osd.5 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:35:08.916 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:35:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:35:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"smithi016:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.16\", device_class=\"ssd\", hostname=\"smithi016\", instance=\"172.21.15.196:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.16\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:35:09.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:08 smithi016 ceph-mon[67348]: pgmap v145: 161 pgs: 22 active+undersized+degraded, 38 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-04-14T15:35:09.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:08 smithi016 ceph-mon[67348]: from='osd.5 [v2:172.21.15.196:6808/2286579913,v1:172.21.15.196:6809/2286579913]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:35:09.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:08 smithi016 ceph-mon[67348]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:35:09.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:08 smithi016 ceph-mon[64807]: pgmap v145: 161 pgs: 22 active+undersized+degraded, 38 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-04-14T15:35:09.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:08 smithi016 ceph-mon[64807]: from='osd.5 [v2:172.21.15.196:6808/2286579913,v1:172.21.15.196:6809/2286579913]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:35:09.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:08 smithi016 ceph-mon[64807]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T15:35:09.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:09 smithi196 ceph-mon[58679]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T15:35:09.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:09 smithi196 ceph-mon[58679]: osdmap e133: 8 total, 7 up, 8 in 2024-04-14T15:35:09.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:09 smithi196 ceph-mon[58679]: from='osd.5 [v2:172.21.15.196:6808/2286579913,v1:172.21.15.196:6809/2286579913]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:09.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:09 smithi196 ceph-mon[58679]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:09.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:09 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:09.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:09 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:10.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[67348]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T15:35:10.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[67348]: osdmap e133: 8 total, 7 up, 8 in 2024-04-14T15:35:10.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[67348]: from='osd.5 [v2:172.21.15.196:6808/2286579913,v1:172.21.15.196:6809/2286579913]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:10.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[67348]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:10.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:10.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:10.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[64807]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T15:35:10.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[64807]: osdmap e133: 8 total, 7 up, 8 in 2024-04-14T15:35:10.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[64807]: from='osd.5 [v2:172.21.15.196:6808/2286579913,v1:172.21.15.196:6809/2286579913]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:10.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[64807]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:10.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:10.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:09 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:10.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:10 smithi196 ceph-mon[58679]: pgmap v147: 161 pgs: 22 active+undersized+degraded, 38 active+undersized, 101 active+clean; 585 KiB data, 204 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-04-14T15:35:10.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:10 smithi196 ceph-mon[58679]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:35:10.937 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:10 smithi196 ceph-mon[58679]: osd.5 [v2:172.21.15.196:6808/2286579913,v1:172.21.15.196:6809/2286579913] boot 2024-04-14T15:35:10.938 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:10 smithi196 ceph-mon[58679]: osdmap e134: 8 total, 8 up, 8 in 2024-04-14T15:35:10.938 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:10 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:35:10.938 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:10 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:10.938 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:10 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:10.938 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:10 smithi196 ceph-mon[58679]: osdmap e135: 8 total, 8 up, 8 in 2024-04-14T15:35:11.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[67348]: pgmap v147: 161 pgs: 22 active+undersized+degraded, 38 active+undersized, 101 active+clean; 585 KiB data, 204 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-04-14T15:35:11.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[67348]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:35:11.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[67348]: osd.5 [v2:172.21.15.196:6808/2286579913,v1:172.21.15.196:6809/2286579913] boot 2024-04-14T15:35:11.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[67348]: osdmap e134: 8 total, 8 up, 8 in 2024-04-14T15:35:11.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:35:11.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:11.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:11.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[67348]: osdmap e135: 8 total, 8 up, 8 in 2024-04-14T15:35:11.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[64807]: pgmap v147: 161 pgs: 22 active+undersized+degraded, 38 active+undersized, 101 active+clean; 585 KiB data, 204 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-04-14T15:35:11.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[64807]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:35:11.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[64807]: osd.5 [v2:172.21.15.196:6808/2286579913,v1:172.21.15.196:6809/2286579913] boot 2024-04-14T15:35:11.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[64807]: osdmap e134: 8 total, 8 up, 8 in 2024-04-14T15:35:11.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T15:35:11.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:11.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:11.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:10 smithi016 ceph-mon[64807]: osdmap e135: 8 total, 8 up, 8 in 2024-04-14T15:35:11.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:11 smithi016 ceph-mon[67348]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 22 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:11.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:11 smithi016 ceph-mon[64807]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 22 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:11.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:35:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:35:11] "GET /metrics HTTP/1.1" 200 38452 "" "Prometheus/2.33.4" 2024-04-14T15:35:11.991 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:11 smithi196 ceph-mon[58679]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 22 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:12.959 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:12 smithi016 ceph-mon[67348]: pgmap v150: 161 pgs: 14 peering, 17 active+undersized+degraded, 29 active+undersized, 101 active+clean; 585 KiB data, 204 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-14T15:35:12.959 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:12.960 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:12.960 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:12 smithi016 ceph-mon[64807]: pgmap v150: 161 pgs: 14 peering, 17 active+undersized+degraded, 29 active+undersized, 101 active+clean; 585 KiB data, 204 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-14T15:35:12.960 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:12.960 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:13.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:12 smithi196 ceph-mon[58679]: pgmap v150: 161 pgs: 14 peering, 17 active+undersized+degraded, 29 active+undersized, 101 active+clean; 585 KiB data, 204 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-04-14T15:35:13.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:12 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:13.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:12 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:13.915 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:35:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:35:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi016\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3ca5917a-fa72-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi016\", job=\"node\", machine=\"x86_64\", nodename=\"smithi016\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T15:35:14.110 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:35:13 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:35:13.633+0000 7fda5603a700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-04-14T15:35:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: pgmap v151: 161 pgs: 14 peering, 12 active+undersized+degraded, 24 active+undersized, 111 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-04-14T15:35:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:35:14.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:35:14.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: pgmap v152: 161 pgs: 14 peering, 12 active+undersized+degraded, 24 active+undersized, 111 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-04-14T15:35:14.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: pgmap v153: 161 pgs: 14 peering, 12 active+undersized+degraded, 24 active+undersized, 111 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-04-14T15:35:14.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:14.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:35:14.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:14.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:14.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:14.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:14.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:14.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[67348]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: pgmap v151: 161 pgs: 14 peering, 12 active+undersized+degraded, 24 active+undersized, 111 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: pgmap v152: 161 pgs: 14 peering, 12 active+undersized+degraded, 24 active+undersized, 111 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: pgmap v153: 161 pgs: 14 peering, 12 active+undersized+degraded, 24 active+undersized, 111 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:14.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:14.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:14.864 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:14 smithi016 ceph-mon[64807]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-04-14T15:35:14.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:14.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: pgmap v151: 161 pgs: 14 peering, 12 active+undersized+degraded, 24 active+undersized, 111 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-04-14T15:35:14.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:14.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:14.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:35:14.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:35:14.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: pgmap v152: 161 pgs: 14 peering, 12 active+undersized+degraded, 24 active+undersized, 111 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-04-14T15:35:14.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: pgmap v153: 161 pgs: 14 peering, 12 active+undersized+degraded, 24 active+undersized, 111 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s; 51/627 objects degraded (8.134%) 2024-04-14T15:35:14.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:14.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:35:14.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:14.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:14.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:14.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:14.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:14.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:14 smithi196 ceph-mon[58679]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-04-14T15:35:15.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:15 smithi016 ceph-mon[64807]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 51/627 objects degraded (8.134%), 12 pgs degraded) 2024-04-14T15:35:15.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:15 smithi016 ceph-mon[67348]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 51/627 objects degraded (8.134%), 12 pgs degraded) 2024-04-14T15:35:15.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:15 smithi196 ceph-mon[58679]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 51/627 objects degraded (8.134%), 12 pgs degraded) 2024-04-14T15:35:16.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:16 smithi016 ceph-mon[64807]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:35:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:16 smithi016 ceph-mon[67348]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:35:16.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:16 smithi196 ceph-mon[58679]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:35:17.594 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:35:18.271 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:35:18.271 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (5m) 62s ago 10m 23.9M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:35:18.271 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 error 9s ago 9m - - 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (4m) 62s ago 9m 52.3M - 3.5 e1d6a67b021e 27d95fa4403f 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (4m) 9s ago 14m 451M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (4m) 62s ago 16m 508M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (3m) 62s ago 16m 48.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (3m) 9s ago 14m 38.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 72817f80f666 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (3m) 62s ago 15m 39.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (5m) 62s ago 10m 17.4M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (4m) 9s ago 10m 18.0M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (2m) 62s ago 14m 44.0M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 f6ee7262903f 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (108s) 62s ago 13m 43.7M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 14dec0c76aaf 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (88s) 62s ago 13m 51.5M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 ce205c8399d3 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (68s) 62s ago 13m 12.5M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 6925155fa41a 2024-04-14T15:35:18.272 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (48s) 9s ago 12m 44.2M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 3a7864fa8601 2024-04-14T15:35:18.273 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (13s) 9s ago 12m 15.0M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 08ddd21e0929 2024-04-14T15:35:18.273 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (12m) 9s ago 12m 55.5M 2992M 17.2.0 e1d6a67b021e cc8841300f97 2024-04-14T15:35:18.273 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (11m) 9s ago 11m 61.3M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:35:18.273 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (4m) 9s ago 10m 73.5M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:35:18.273 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (9m) 62s ago 9m 75.3M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:35:18.273 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (9m) 9s ago 9m 75.6M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:35:18.273 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (9m) 62s ago 9m 73.2M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:35:18.273 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (9m) 9s ago 9m 76.0M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:35:18.682 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:35:18.682 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:35:18.682 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:35:18.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:35:18.682 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:35:18.682 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:35:18.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:35:18.682 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:35:18.682 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-14T15:35:18.682 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 6 2024-04-14T15:35:18.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:35:18.683 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:35:18.683 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:35:18.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:35:18.683 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:35:18.683 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-14T15:35:18.683 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 11 2024-04-14T15:35:18.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:35:18.683 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:35:18.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:18 smithi016 ceph-mon[67348]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 751 B/s rd, 0 op/s 2024-04-14T15:35:18.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:18 smithi016 ceph-mon[67348]: from='client.54346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:18.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:18 smithi016 ceph-mon[64807]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 751 B/s rd, 0 op/s 2024-04-14T15:35:18.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:18 smithi016 ceph-mon[64807]: from='client.54346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:18.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:18 smithi196 ceph-mon[58679]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 751 B/s rd, 0 op/s 2024-04-14T15:35:18.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:18 smithi196 ceph-mon[58679]: from='client.54346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:19.030 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:35:19.030 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:35:19.030 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:35:19.030 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:35:19.031 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:35:19.031 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-04-14T15:35:19.031 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-14T15:35:19.031 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:35:19.031 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "11/23 daemons upgraded", 2024-04-14T15:35:19.031 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T15:35:19.031 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:35:19.031 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:35:19.440 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:35:19.441 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:35:19.441 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in error state 2024-04-14T15:35:19.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:19 smithi016 ceph-mon[67348]: from='client.44385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:19.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:19 smithi016 ceph-mon[67348]: from='client.54358 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:19.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:19 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/246192573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:19.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:19 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/2707255907' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:35:19.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:19 smithi016 ceph-mon[64807]: from='client.44385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:19.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:19 smithi016 ceph-mon[64807]: from='client.54358 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:19.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:19 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/246192573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:19.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:19 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/2707255907' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:35:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:19 smithi196 ceph-mon[58679]: from='client.44385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:19 smithi196 ceph-mon[58679]: from='client.54358 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:19 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/246192573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:19 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/2707255907' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:35:20.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:20 smithi016 ceph-mon[64807]: from='client.44403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:20.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:20 smithi016 ceph-mon[64807]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-14T15:35:20.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:20 smithi016 ceph-mon[67348]: from='client.44403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:20.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:20 smithi016 ceph-mon[67348]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-14T15:35:20.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:20 smithi196 ceph-mon[58679]: from='client.44403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:20.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:20 smithi196 ceph-mon[58679]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 624 B/s rd, 0 op/s 2024-04-14T15:35:21.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:35:21 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:35:21] "GET /metrics HTTP/1.1" 200 38322 "" "Prometheus/2.33.4" 2024-04-14T15:35:22.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:22 smithi016 ceph-mon[67348]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:22.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:22 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:22.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:22 smithi016 ceph-mon[64807]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:22.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:22 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:22.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:22 smithi196 ceph-mon[58679]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:22.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:22 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:24.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:24 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:24.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:24 smithi016 ceph-mon[64807]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:35:24.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:24 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:24.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:24 smithi016 ceph-mon[67348]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:35:24.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:24 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:24.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:24 smithi196 ceph-mon[58679]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:35:26.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:26 smithi016 ceph-mon[67348]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:26.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:26 smithi016 ceph-mon[64807]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:26 smithi196 ceph-mon[58679]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:27 smithi196 ceph-mon[58679]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:28.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:28.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:28.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:27 smithi016 ceph-mon[67348]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:28.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:28.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:28.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:27 smithi016 ceph-mon[64807]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:29.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:28 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:29.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:28 smithi196 ceph-mon[58679]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:29.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:28 smithi196 ceph-mon[58679]: Upgrade: osd.6 is safe to restart 2024-04-14T15:35:29.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:28 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:29.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:28 smithi016 ceph-mon[67348]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:29.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:28 smithi016 ceph-mon[67348]: Upgrade: osd.6 is safe to restart 2024-04-14T15:35:29.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:28 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:29.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:28 smithi016 ceph-mon[64807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T15:35:29.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:28 smithi016 ceph-mon[64807]: Upgrade: osd.6 is safe to restart 2024-04-14T15:35:30.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:29 smithi196 ceph-mon[58679]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:30.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:29 smithi016 ceph-mon[67348]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:30.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:29 smithi016 ceph-mon[64807]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:31 smithi196 ceph-mon[58679]: Upgrade: Updating osd.6 2024-04-14T15:35:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:31 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:31.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:31 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:35:31.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:31 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:35:31.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:31 smithi196 ceph-mon[58679]: Deploying daemon osd.6 on smithi196 2024-04-14T15:35:31.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:35:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:35:31] "GET /metrics HTTP/1.1" 200 38466 "" "Prometheus/2.33.4" 2024-04-14T15:35:31.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:31 smithi016 ceph-mon[67348]: Upgrade: Updating osd.6 2024-04-14T15:35:31.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:31 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:31.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:31 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:35:31.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:31 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:35:31.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:31 smithi016 ceph-mon[67348]: Deploying daemon osd.6 on smithi196 2024-04-14T15:35:31.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:31 smithi016 ceph-mon[64807]: Upgrade: Updating osd.6 2024-04-14T15:35:31.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:31 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:31.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:31 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T15:35:31.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:31 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:35:31.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:31 smithi016 ceph-mon[64807]: Deploying daemon osd.6 on smithi196 2024-04-14T15:35:32.543 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:32 smithi196 ceph-mon[58679]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:32.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:32 smithi016 ceph-mon[67348]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:32.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:32 smithi016 ceph-mon[64807]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:33.902 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:33 smithi196 systemd[1]: Stopping Ceph osd.6 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:35:34.165 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[34914]: 2024-04-14T15:35:33.901+0000 7f8062d38700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:35:34.165 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[34914]: 2024-04-14T15:35:33.901+0000 7f8062d38700 -1 osd.6 135 *** Got signal Terminated *** 2024-04-14T15:35:34.165 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:33 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[34914]: 2024-04-14T15:35:33.901+0000 7f8062d38700 -1 osd.6 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:35:34.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:34 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:34.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:34 smithi016 ceph-mon[64807]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:34.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:34 smithi016 ceph-mon[64807]: osd.6 marked itself down and dead 2024-04-14T15:35:34.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:34 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:34.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:34 smithi016 ceph-mon[67348]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:34.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:34 smithi016 ceph-mon[67348]: osd.6 marked itself down and dead 2024-04-14T15:35:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:34 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:34 smithi196 ceph-mon[58679]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:35:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:34 smithi196 ceph-mon[58679]: osd.6 marked itself down and dead 2024-04-14T15:35:35.261 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:34 smithi196 podman[66430]: 2024-04-14 15:35:34.99557799 +0000 UTC m=+1.201057883 container died cc8841300f9776ce440247d75ae647dc7f41ad31de75ca202505e93f2ae1cebd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, version=8, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , release=754, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, distribution-scope=public, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream) 2024-04-14T15:35:35.569 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:35 smithi196 podman[66430]: 2024-04-14 15:35:35.426424111 +0000 UTC m=+1.631904071 container cleanup cc8841300f9776ce440247d75ae647dc7f41ad31de75ca202505e93f2ae1cebd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, vcs-type=git, distribution-scope=public, release=754, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., version=8, maintainer=Guillaume Abrioux ) 2024-04-14T15:35:35.569 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:35 smithi196 bash[66430]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6 2024-04-14T15:35:35.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:35 smithi016 ceph-mon[67348]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:35:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:35 smithi016 ceph-mon[67348]: osdmap e136: 8 total, 7 up, 8 in 2024-04-14T15:35:35.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:35 smithi016 ceph-mon[67348]: osdmap e137: 8 total, 7 up, 8 in 2024-04-14T15:35:35.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:35 smithi016 ceph-mon[64807]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:35:35.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:35 smithi016 ceph-mon[64807]: osdmap e136: 8 total, 7 up, 8 in 2024-04-14T15:35:35.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:35 smithi016 ceph-mon[64807]: osdmap e137: 8 total, 7 up, 8 in 2024-04-14T15:35:35.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:35 smithi196 ceph-mon[58679]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:35:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:35 smithi196 ceph-mon[58679]: osdmap e136: 8 total, 7 up, 8 in 2024-04-14T15:35:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:35 smithi196 ceph-mon[58679]: osdmap e137: 8 total, 7 up, 8 in 2024-04-14T15:35:35.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:35 smithi196 podman[66447]: 2024-04-14 15:35:35.636361217 +0000 UTC m=+0.638881455 container remove cc8841300f9776ce440247d75ae647dc7f41ad31de75ca202505e93f2ae1cebd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_CLEAN=True, architecture=x86_64, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, name=centos-stream, vendor=Red Hat, Inc., vcs-type=git, GIT_BRANCH=HEAD, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-14T15:35:36.383 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:36 smithi196 podman[66539]: 2024-04-14 15:35:35.943681672 +0000 UTC m=+0.017716196 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:35:36.383 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:36 smithi196 podman[66539]: 2024-04-14 15:35:36.053679058 +0000 UTC m=+0.127713580 container create 366570588b86c0982a809da02e18698a0907bc3cefb56365916ded9c971a867c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:35:36.383 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:36 smithi196 podman[66539]: 2024-04-14 15:35:36.283386157 +0000 UTC m=+0.357420679 container init 366570588b86c0982a809da02e18698a0907bc3cefb56365916ded9c971a867c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD) 2024-04-14T15:35:36.383 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:36 smithi196 podman[66539]: 2024-04-14 15:35:36.286661563 +0000 UTC m=+0.360696086 container start 366570588b86c0982a809da02e18698a0907bc3cefb56365916ded9c971a867c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-14T15:35:36.383 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:36 smithi196 podman[66539]: 2024-04-14 15:35:36.383273091 +0000 UTC m=+0.457307624 container attach 366570588b86c0982a809da02e18698a0907bc3cefb56365916ded9c971a867c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-14T15:35:36.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:36 smithi196 ceph-mon[58679]: pgmap v165: 161 pgs: 26 peering, 6 stale+active+clean, 129 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:35:36.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:36 smithi196 ceph-mon[58679]: Health check failed: Reduced data availability: 7 pgs peering (PG_AVAILABILITY) 2024-04-14T15:35:36.665 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:36 smithi196 podman[66539]: 2024-04-14 15:35:36.577900166 +0000 UTC m=+0.651934693 container died 366570588b86c0982a809da02e18698a0907bc3cefb56365916ded9c971a867c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, RELEASE=main-a971fd5, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-14T15:35:36.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:36 smithi016 ceph-mon[67348]: pgmap v165: 161 pgs: 26 peering, 6 stale+active+clean, 129 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:35:36.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:36 smithi016 ceph-mon[67348]: Health check failed: Reduced data availability: 7 pgs peering (PG_AVAILABILITY) 2024-04-14T15:35:36.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:36 smithi016 ceph-mon[64807]: pgmap v165: 161 pgs: 26 peering, 6 stale+active+clean, 129 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:35:36.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:36 smithi016 ceph-mon[64807]: Health check failed: Reduced data availability: 7 pgs peering (PG_AVAILABILITY) 2024-04-14T15:35:37.417 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:37 smithi196 podman[66566]: 2024-04-14 15:35:37.145025109 +0000 UTC m=+0.562203949 container remove 366570588b86c0982a809da02e18698a0907bc3cefb56365916ded9c971a867c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-14T15:35:37.417 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:37 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.6.service: Deactivated successfully. 2024-04-14T15:35:37.418 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:37 smithi196 systemd[1]: Stopped Ceph osd.6 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:35:37.418 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:37 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.6.service: Consumed 7.547s CPU time. 2024-04-14T15:35:37.418 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:37 smithi196 systemd[1]: Starting Ceph osd.6 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:35:37.874 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:37 smithi196 podman[66669]: 2024-04-14 15:35:37.48233366 +0000 UTC m=+0.017985817 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:35:37.874 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:37 smithi196 podman[66669]: 2024-04-14 15:35:37.587615408 +0000 UTC m=+0.123267558 container create b5312f97df9507cfd62f58b993a5c2b8e905401d45face1950071d9062d3af67 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:35:37.874 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:37 smithi196 podman[66669]: 2024-04-14 15:35:37.874233683 +0000 UTC m=+0.409885832 container init b5312f97df9507cfd62f58b993a5c2b8e905401d45face1950071d9062d3af67 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS) 2024-04-14T15:35:38.165 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:37 smithi196 podman[66669]: 2024-04-14 15:35:37.877697152 +0000 UTC m=+0.413349304 container start b5312f97df9507cfd62f58b993a5c2b8e905401d45face1950071d9062d3af67 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:35:38.165 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:37 smithi196 podman[66669]: 2024-04-14 15:35:37.948795578 +0000 UTC m=+0.484447737 container attach b5312f97df9507cfd62f58b993a5c2b8e905401d45face1950071d9062d3af67 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-a971fd5) 2024-04-14T15:35:38.573 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate[66683]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate[66683]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate[66683]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate[66683]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate[66683]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate[66683]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 bash[66669]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 bash[66669]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 bash[66669]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 bash[66669]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 bash[66669]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 bash[66669]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 bash[66669]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-14T15:35:38.574 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate[66683]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-14T15:35:38.575 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 podman[66669]: 2024-04-14 15:35:38.343476292 +0000 UTC m=+0.879128433 container died b5312f97df9507cfd62f58b993a5c2b8e905401d45face1950071d9062d3af67 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-14T15:35:38.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:38 smithi016 ceph-mon[67348]: pgmap v167: 161 pgs: 26 peering, 6 stale+active+clean, 129 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:35:38.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:38 smithi016 ceph-mon[64807]: pgmap v167: 161 pgs: 26 peering, 6 stale+active+clean, 129 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:35:38.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:38 smithi196 ceph-mon[58679]: pgmap v167: 161 pgs: 26 peering, 6 stale+active+clean, 129 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:35:39.203 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:38 smithi196 podman[66805]: 2024-04-14 15:35:38.952653157 +0000 UTC m=+0.604104160 container remove b5312f97df9507cfd62f58b993a5c2b8e905401d45face1950071d9062d3af67 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5) 2024-04-14T15:35:39.203 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:39 smithi196 podman[66871]: 2024-04-14 15:35:39.074464157 +0000 UTC m=+0.017790603 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:35:39.203 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:39 smithi196 podman[66871]: 2024-04-14 15:35:39.203609607 +0000 UTC m=+0.146936039 container create 31fa3bea2859b81804bd4334af7826b6a26b6fda6ae86dbdba25b56ac4588622 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-14T15:35:39.471 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:39 smithi196 podman[66871]: 2024-04-14 15:35:39.448255609 +0000 UTC m=+0.391582047 container init 31fa3bea2859b81804bd4334af7826b6a26b6fda6ae86dbdba25b56ac4588622 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:35:39.471 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:39 smithi196 podman[66871]: 2024-04-14 15:35:39.451575631 +0000 UTC m=+0.394902071 container start 31fa3bea2859b81804bd4334af7826b6a26b6fda6ae86dbdba25b56ac4588622 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-14T15:35:39.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:39 smithi196 ceph-mon[58679]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 10 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:39.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:39 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:39.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:39 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:39.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:39 smithi196 ceph-osd[66889]: -- 172.21.15.196:0/398181592 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55edae5f1680 con 0x55edae5bb000 2024-04-14T15:35:39.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:39 smithi196 bash[66871]: 31fa3bea2859b81804bd4334af7826b6a26b6fda6ae86dbdba25b56ac4588622 2024-04-14T15:35:39.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:39 smithi196 systemd[1]: Started Ceph osd.6 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:35:40.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:39 smithi016 ceph-mon[67348]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 10 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:40.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:39 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:40.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:39 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:39 smithi016 ceph-mon[64807]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 10 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:39 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:39 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:40.415 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:39 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:35:39.993+0000 7fe5e16537c0 -1 Falling back to public interface 2024-04-14T15:35:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:40 smithi196 ceph-mon[58679]: pgmap v168: 161 pgs: 10 active+undersized+degraded, 26 peering, 17 active+undersized, 108 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-04-14T15:35:41.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:40 smithi016 ceph-mon[67348]: pgmap v168: 161 pgs: 10 active+undersized+degraded, 26 peering, 17 active+undersized, 108 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-04-14T15:35:41.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:40 smithi016 ceph-mon[64807]: pgmap v168: 161 pgs: 10 active+undersized+degraded, 26 peering, 17 active+undersized, 108 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-04-14T15:35:41.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:41 smithi016 ceph-mon[64807]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-04-14T15:35:41.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:35:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:35:41] "GET /metrics HTTP/1.1" 200 38466 "" "Prometheus/2.33.4" 2024-04-14T15:35:41.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:41 smithi016 ceph-mon[67348]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-04-14T15:35:41.964 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:41 smithi196 ceph-mon[58679]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-04-14T15:35:42.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:42 smithi196 ceph-mon[58679]: pgmap v169: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-04-14T15:35:42.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:42.936 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:42.936 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:42 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:35:42.704+0000 7fe5e16537c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-04-14T15:35:42.936 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:42 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:35:42.733+0000 7fe5e16537c0 -1 osd.6 135 log_to_monitors true 2024-04-14T15:35:42.988 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:42 smithi016 ceph-mon[67348]: pgmap v169: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-04-14T15:35:42.988 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:42.988 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:42.988 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:42 smithi016 ceph-mon[64807]: pgmap v169: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-04-14T15:35:42.988 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:42.989 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:43.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:35:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:35:43.711+0000 7fe5d2aab700 -1 osd.6 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:35:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:43 smithi196 ceph-mon[58679]: from='osd.6 [v2:172.21.15.196:6816/1725788136,v1:172.21.15.196:6817/1725788136]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:35:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:43 smithi196 ceph-mon[58679]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:35:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:43 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:43 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:44.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:43 smithi016 ceph-mon[67348]: from='osd.6 [v2:172.21.15.196:6816/1725788136,v1:172.21.15.196:6817/1725788136]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:35:44.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:43 smithi016 ceph-mon[67348]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:35:44.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:43 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:44.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:43 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:44.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:43 smithi016 ceph-mon[64807]: from='osd.6 [v2:172.21.15.196:6816/1725788136,v1:172.21.15.196:6817/1725788136]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:35:44.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:43 smithi016 ceph-mon[64807]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T15:35:44.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:43 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:44.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:43 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:45.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:45.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[64807]: pgmap v170: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-04-14T15:35:45.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[64807]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T15:35:45.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[64807]: osdmap e138: 8 total, 7 up, 8 in 2024-04-14T15:35:45.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[64807]: from='osd.6 [v2:172.21.15.196:6816/1725788136,v1:172.21.15.196:6817/1725788136]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:45.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[64807]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:45.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:45.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:45.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:45.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[67348]: pgmap v170: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-04-14T15:35:45.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[67348]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T15:35:45.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[67348]: osdmap e138: 8 total, 7 up, 8 in 2024-04-14T15:35:45.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[67348]: from='osd.6 [v2:172.21.15.196:6816/1725788136,v1:172.21.15.196:6817/1725788136]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:45.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[67348]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:45.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:45.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:44 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:45.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:44 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:45.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:44 smithi196 ceph-mon[58679]: pgmap v170: 161 pgs: 20 active+undersized+degraded, 33 active+undersized, 108 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-04-14T15:35:45.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:44 smithi196 ceph-mon[58679]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T15:35:45.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:44 smithi196 ceph-mon[58679]: osdmap e138: 8 total, 7 up, 8 in 2024-04-14T15:35:45.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:44 smithi196 ceph-mon[58679]: from='osd.6 [v2:172.21.15.196:6816/1725788136,v1:172.21.15.196:6817/1725788136]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:45.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:44 smithi196 ceph-mon[58679]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:35:45.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:44 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:45.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:44 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:45 smithi196 ceph-mon[58679]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:35:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:45 smithi196 ceph-mon[58679]: osd.6 [v2:172.21.15.196:6816/1725788136,v1:172.21.15.196:6817/1725788136] boot 2024-04-14T15:35:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:45 smithi196 ceph-mon[58679]: osdmap e139: 8 total, 8 up, 8 in 2024-04-14T15:35:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:45 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:35:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:45 smithi196 ceph-mon[58679]: osdmap e140: 8 total, 8 up, 8 in 2024-04-14T15:35:46.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:45 smithi016 ceph-mon[67348]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:35:46.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:45 smithi016 ceph-mon[67348]: osd.6 [v2:172.21.15.196:6816/1725788136,v1:172.21.15.196:6817/1725788136] boot 2024-04-14T15:35:46.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:45 smithi016 ceph-mon[67348]: osdmap e139: 8 total, 8 up, 8 in 2024-04-14T15:35:46.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:45 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:35:46.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:45 smithi016 ceph-mon[67348]: osdmap e140: 8 total, 8 up, 8 in 2024-04-14T15:35:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:45 smithi016 ceph-mon[64807]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:35:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:45 smithi016 ceph-mon[64807]: osd.6 [v2:172.21.15.196:6816/1725788136,v1:172.21.15.196:6817/1725788136] boot 2024-04-14T15:35:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:45 smithi016 ceph-mon[64807]: osdmap e139: 8 total, 8 up, 8 in 2024-04-14T15:35:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:45 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T15:35:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:45 smithi016 ceph-mon[64807]: osdmap e140: 8 total, 8 up, 8 in 2024-04-14T15:35:46.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:46 smithi196 ceph-mon[58679]: pgmap v173: 161 pgs: 19 peering, 13 active+undersized+degraded, 21 active+undersized, 108 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 50/627 objects degraded (7.974%) 2024-04-14T15:35:46.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:46 smithi196 ceph-mon[58679]: Health check update: Degraded data redundancy: 50/627 objects degraded (7.974%), 13 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:47.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:46 smithi016 ceph-mon[67348]: pgmap v173: 161 pgs: 19 peering, 13 active+undersized+degraded, 21 active+undersized, 108 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 50/627 objects degraded (7.974%) 2024-04-14T15:35:47.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:46 smithi016 ceph-mon[67348]: Health check update: Degraded data redundancy: 50/627 objects degraded (7.974%), 13 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:47.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:46 smithi016 ceph-mon[64807]: pgmap v173: 161 pgs: 19 peering, 13 active+undersized+degraded, 21 active+undersized, 108 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 50/627 objects degraded (7.974%) 2024-04-14T15:35:47.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:46 smithi016 ceph-mon[64807]: Health check update: Degraded data redundancy: 50/627 objects degraded (7.974%), 13 pgs degraded (PG_DEGRADED) 2024-04-14T15:35:48.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:35:48.234+0000 7fda5603a700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-14T15:35:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[64807]: pgmap v175: 161 pgs: 19 peering, 13 active+undersized+degraded, 21 active+undersized, 108 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 50/627 objects degraded (7.974%) 2024-04-14T15:35:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:35:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:35:49.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[67348]: pgmap v175: 161 pgs: 19 peering, 13 active+undersized+degraded, 21 active+undersized, 108 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 50/627 objects degraded (7.974%) 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:49.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:35:49.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:49.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:49.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:49.113 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:35:49.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:48 smithi196 ceph-mon[58679]: pgmap v175: 161 pgs: 19 peering, 13 active+undersized+degraded, 21 active+undersized, 108 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 50/627 objects degraded (7.974%) 2024-04-14T15:35:49.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:49.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:49.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:35:49.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:35:49.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:49.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:35:49.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:49.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:49.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:49.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:35:49.813 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:35:50.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:49 smithi016 ceph-mon[67348]: pgmap v176: 161 pgs: 19 peering, 10 active+undersized+degraded, 15 active+undersized, 117 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s; 35/627 objects degraded (5.582%) 2024-04-14T15:35:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:49 smithi016 ceph-mon[67348]: pgmap v177: 161 pgs: 19 peering, 10 active+undersized+degraded, 15 active+undersized, 117 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s; 35/627 objects degraded (5.582%) 2024-04-14T15:35:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:49 smithi016 ceph-mon[67348]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:35:50.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:49 smithi016 ceph-mon[67348]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-14T15:35:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:49 smithi016 ceph-mon[64807]: pgmap v176: 161 pgs: 19 peering, 10 active+undersized+degraded, 15 active+undersized, 117 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s; 35/627 objects degraded (5.582%) 2024-04-14T15:35:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:49 smithi016 ceph-mon[64807]: pgmap v177: 161 pgs: 19 peering, 10 active+undersized+degraded, 15 active+undersized, 117 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s; 35/627 objects degraded (5.582%) 2024-04-14T15:35:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:49 smithi016 ceph-mon[64807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:35:50.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:49 smithi016 ceph-mon[64807]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-14T15:35:50.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:49 smithi196 ceph-mon[58679]: pgmap v176: 161 pgs: 19 peering, 10 active+undersized+degraded, 15 active+undersized, 117 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s; 35/627 objects degraded (5.582%) 2024-04-14T15:35:50.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:49 smithi196 ceph-mon[58679]: pgmap v177: 161 pgs: 19 peering, 10 active+undersized+degraded, 15 active+undersized, 117 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s; 35/627 objects degraded (5.582%) 2024-04-14T15:35:50.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:49 smithi196 ceph-mon[58679]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:35:50.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:49 smithi196 ceph-mon[58679]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-14T15:35:50.492 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:35:50.492 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (5m) 95s ago 10m 23.9M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:35:50.492 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 error 6s ago 10m - - 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (4m) 95s ago 9m 52.3M - 3.5 e1d6a67b021e 27d95fa4403f 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (4m) 6s ago 15m 452M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (5m) 95s ago 16m 508M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (4m) 95s ago 16m 48.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (3m) 6s ago 15m 40.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 72817f80f666 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (3m) 95s ago 15m 39.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (5m) 95s ago 11m 17.4M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (5m) 6s ago 11m 17.9M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (2m) 95s ago 14m 44.0M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 f6ee7262903f 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (2m) 95s ago 14m 43.7M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 14dec0c76aaf 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (2m) 95s ago 14m 51.5M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 ce205c8399d3 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (100s) 95s ago 13m 12.5M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 6925155fa41a 2024-04-14T15:35:50.493 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (80s) 6s ago 13m 44.8M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 3a7864fa8601 2024-04-14T15:35:50.494 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (45s) 6s ago 12m 45.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 08ddd21e0929 2024-04-14T15:35:50.494 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (11s) 6s ago 12m 14.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 31fa3bea2859 2024-04-14T15:35:50.494 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (12m) 6s ago 12m 61.9M 2992M 17.2.0 e1d6a67b021e b7466f0450dd 2024-04-14T15:35:50.494 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (4m) 6s ago 11m 73.5M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:35:50.494 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (9m) 95s ago 9m 75.3M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:35:50.494 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (9m) 6s ago 9m 75.7M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:35:50.494 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (9m) 95s ago 9m 73.2M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:35:50.494 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (9m) 6s ago 9m 76.2M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:35:50.911 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:35:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:35:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:35:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:35:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:35:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:35:50.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 7 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 12 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:35:50.912 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:35:51.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:50 smithi016 ceph-mon[67348]: from='client.44415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:51.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:50 smithi016 ceph-mon[67348]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:35:51.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:50 smithi016 ceph-mon[67348]: from='client.54388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:51.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:50 smithi016 ceph-mon[67348]: from='client.44427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:51.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:50 smithi016 ceph-mon[67348]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/627 objects degraded (5.582%), 10 pgs degraded) 2024-04-14T15:35:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:50 smithi016 ceph-mon[64807]: from='client.44415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:50 smithi016 ceph-mon[64807]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:35:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:50 smithi016 ceph-mon[64807]: from='client.54388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:50 smithi016 ceph-mon[64807]: from='client.44427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:50 smithi016 ceph-mon[64807]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/627 objects degraded (5.582%), 10 pgs degraded) 2024-04-14T15:35:51.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:50 smithi196 ceph-mon[58679]: from='client.44415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:51.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:50 smithi196 ceph-mon[58679]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T15:35:51.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:50 smithi196 ceph-mon[58679]: from='client.54388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:51.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:50 smithi196 ceph-mon[58679]: from='client.44427 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:51.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:50 smithi196 ceph-mon[58679]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/627 objects degraded (5.582%), 10 pgs degraded) 2024-04-14T15:35:51.256 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:35:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:35:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:35:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:35:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:35:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-04-14T15:35:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr" 2024-04-14T15:35:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:35:51.256 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "12/23 daemons upgraded", 2024-04-14T15:35:51.257 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T15:35:51.257 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:35:51.257 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:35:51.671 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:35:51.671 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:35:51.671 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in error state 2024-04-14T15:35:51.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:35:51 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:35:51] "GET /metrics HTTP/1.1" 200 38466 "" "Prometheus/2.33.4" 2024-04-14T15:35:51.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:51 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/1580591844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:51.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:51 smithi016 ceph-mon[67348]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:51.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:51 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/1254753765' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:35:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:51 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/1580591844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:51 smithi016 ceph-mon[64807]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:51.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:51 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/1254753765' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:35:52.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:51 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/1580591844' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:35:52.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:51 smithi196 ceph-mon[58679]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:35:52.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:51 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/1254753765' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:35:53.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:52 smithi016 ceph-mon[64807]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 776 B/s rd, 0 op/s 2024-04-14T15:35:53.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:52 smithi016 ceph-mon[67348]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 776 B/s rd, 0 op/s 2024-04-14T15:35:53.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:52 smithi196 ceph-mon[58679]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 776 B/s rd, 0 op/s 2024-04-14T15:35:54.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:53 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:54.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:53 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:54.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:53 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:35:55.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:54 smithi016 ceph-mon[67348]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2024-04-14T15:35:55.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:54 smithi016 ceph-mon[64807]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2024-04-14T15:35:55.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:54 smithi196 ceph-mon[58679]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2024-04-14T15:35:57.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:57 smithi196 ceph-mon[58679]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:57.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:57.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:57.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:57 smithi016 ceph-mon[64807]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:57.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:57.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:57 smithi016 ceph-mon[67348]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:35:57.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:57.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:35:57.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:35:59.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:35:59 smithi196 ceph-mon[58679]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-14T15:35:59.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:35:59 smithi016 ceph-mon[64807]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-14T15:35:59.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:35:59 smithi016 ceph-mon[67348]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-14T15:36:01.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:01 smithi196 ceph-mon[58679]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-04-14T15:36:01.567 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:01 smithi016 ceph-mon[67348]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-04-14T15:36:01.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:01 smithi016 ceph-mon[64807]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-04-14T15:36:01.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:36:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:36:01] "GET /metrics HTTP/1.1" 200 38477 "" "Prometheus/2.33.4" 2024-04-14T15:36:03.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:03 smithi016 ceph-mon[67348]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-04-14T15:36:03.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:03 smithi016 ceph-mon[64807]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-04-14T15:36:03.414 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:03 smithi196 ceph-mon[58679]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-04-14T15:36:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:04 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:04 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:36:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:04 smithi196 ceph-mon[58679]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:36:04.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:04 smithi196 ceph-mon[58679]: Upgrade: osd.7 is safe to restart 2024-04-14T15:36:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:04 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:04 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:36:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:04 smithi016 ceph-mon[67348]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:36:04.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:04 smithi016 ceph-mon[67348]: Upgrade: osd.7 is safe to restart 2024-04-14T15:36:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:04 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:04 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:36:04.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:04 smithi016 ceph-mon[64807]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T15:36:04.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:04 smithi016 ceph-mon[64807]: Upgrade: osd.7 is safe to restart 2024-04-14T15:36:05.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:05 smithi196 ceph-mon[58679]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-04-14T15:36:05.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:05 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:05.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:05 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:36:05.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:05 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:05.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:05 smithi016 ceph-mon[67348]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-04-14T15:36:05.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:05 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:05.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:05 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:36:05.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:05 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:05.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:05 smithi016 ceph-mon[64807]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-04-14T15:36:05.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:05 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:05.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:05 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T15:36:05.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:05 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:06 smithi196 ceph-mon[58679]: Upgrade: Updating osd.7 2024-04-14T15:36:06.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:06 smithi196 ceph-mon[58679]: Deploying daemon osd.7 on smithi196 2024-04-14T15:36:06.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:06 smithi016 ceph-mon[67348]: Upgrade: Updating osd.7 2024-04-14T15:36:06.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:06 smithi016 ceph-mon[67348]: Deploying daemon osd.7 on smithi196 2024-04-14T15:36:06.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:06 smithi016 ceph-mon[64807]: Upgrade: Updating osd.7 2024-04-14T15:36:06.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:06 smithi016 ceph-mon[64807]: Deploying daemon osd.7 on smithi196 2024-04-14T15:36:07.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:07 smithi196 ceph-mon[58679]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-04-14T15:36:07.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:07 smithi016 ceph-mon[67348]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-04-14T15:36:07.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:07 smithi016 ceph-mon[64807]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 0 B/s wr, 146 op/s 2024-04-14T15:36:08.415 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:08 smithi196 systemd[1]: Stopping Ceph osd.7 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:36:08.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[37913]: 2024-04-14T15:36:08.601+0000 7fdf5cd93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:36:08.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[37913]: 2024-04-14T15:36:08.601+0000 7fdf5cd93700 -1 osd.7 140 *** Got signal Terminated *** 2024-04-14T15:36:08.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[37913]: 2024-04-14T15:36:08.601+0000 7fdf5cd93700 -1 osd.7 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:36:09.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:09 smithi196 ceph-mon[58679]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 169 op/s 2024-04-14T15:36:09.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:09 smithi196 ceph-mon[58679]: osd.7 marked itself down and dead 2024-04-14T15:36:09.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:09 smithi016 ceph-mon[67348]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 169 op/s 2024-04-14T15:36:09.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:09 smithi016 ceph-mon[67348]: osd.7 marked itself down and dead 2024-04-14T15:36:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:09 smithi016 ceph-mon[64807]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 0 B/s wr, 169 op/s 2024-04-14T15:36:09.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:09 smithi016 ceph-mon[64807]: osd.7 marked itself down and dead 2024-04-14T15:36:09.995 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:09 smithi196 podman[68714]: 2024-04-14 15:36:09.705858477 +0000 UTC m=+1.251035916 container died b7466f0450ddc61d3ee0d8a6203d3cccd3dfac69015c1e8408a14047550836eb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, name=centos-stream, distribution-scope=public, vcs-type=git, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, ceph=True) 2024-04-14T15:36:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:10 smithi196 ceph-mon[58679]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:36:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:10 smithi196 ceph-mon[58679]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-14T15:36:10.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:10 smithi196 ceph-mon[58679]: osdmap e141: 8 total, 7 up, 8 in 2024-04-14T15:36:10.415 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:10 smithi196 podman[68714]: 2024-04-14 15:36:10.186400504 +0000 UTC m=+1.731577934 container cleanup b7466f0450ddc61d3ee0d8a6203d3cccd3dfac69015c1e8408a14047550836eb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7, version=8, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0) 2024-04-14T15:36:10.415 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:10 smithi196 bash[68714]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7 2024-04-14T15:36:10.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:10 smithi016 ceph-mon[64807]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:36:10.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:10 smithi016 ceph-mon[64807]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-14T15:36:10.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:10 smithi016 ceph-mon[64807]: osdmap e141: 8 total, 7 up, 8 in 2024-04-14T15:36:10.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:10 smithi016 ceph-mon[67348]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T15:36:10.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:10 smithi016 ceph-mon[67348]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-14T15:36:10.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:10 smithi016 ceph-mon[67348]: osdmap e141: 8 total, 7 up, 8 in 2024-04-14T15:36:10.795 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:10 smithi196 podman[68732]: 2024-04-14 15:36:10.462139744 +0000 UTC m=+0.750917428 container remove b7466f0450ddc61d3ee0d8a6203d3cccd3dfac69015c1e8408a14047550836eb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_CLEAN=True, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.buildah.version=1.19.8, vendor=Red Hat, Inc., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, version=8) 2024-04-14T15:36:11.123 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:10 smithi196 podman[68827]: 2024-04-14 15:36:10.698689161 +0000 UTC m=+0.018099051 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:36:11.123 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:10 smithi196 podman[68827]: 2024-04-14 15:36:10.821466704 +0000 UTC m=+0.140876581 container create 8309d618722f1466788f74d9ebc5be57d0428a7a8f2a7a3c77487b72ba7b49a6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:36:11.124 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:11 smithi196 podman[68827]: 2024-04-14 15:36:11.040553066 +0000 UTC m=+0.359962946 container init 8309d618722f1466788f74d9ebc5be57d0428a7a8f2a7a3c77487b72ba7b49a6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-14T15:36:11.124 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:11 smithi196 podman[68827]: 2024-04-14 15:36:11.043753279 +0000 UTC m=+0.363163159 container start 8309d618722f1466788f74d9ebc5be57d0428a7a8f2a7a3c77487b72ba7b49a6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-14T15:36:11.124 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:11 smithi196 podman[68827]: 2024-04-14 15:36:11.123696141 +0000 UTC m=+0.443106021 container attach 8309d618722f1466788f74d9ebc5be57d0428a7a8f2a7a3c77487b72ba7b49a6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD) 2024-04-14T15:36:11.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:11 smithi196 ceph-mon[58679]: pgmap v189: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 0 B/s wr, 217 op/s 2024-04-14T15:36:11.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:11 smithi196 ceph-mon[58679]: Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-04-14T15:36:11.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:11 smithi196 ceph-mon[58679]: osdmap e142: 8 total, 7 up, 8 in 2024-04-14T15:36:11.415 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:11 smithi196 podman[68827]: 2024-04-14 15:36:11.335886845 +0000 UTC m=+0.655296716 container died 8309d618722f1466788f74d9ebc5be57d0428a7a8f2a7a3c77487b72ba7b49a6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-14T15:36:11.567 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:11 smithi016 ceph-mon[67348]: pgmap v189: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 0 B/s wr, 217 op/s 2024-04-14T15:36:11.567 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:11 smithi016 ceph-mon[67348]: Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-04-14T15:36:11.568 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:11 smithi016 ceph-mon[67348]: osdmap e142: 8 total, 7 up, 8 in 2024-04-14T15:36:11.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:11 smithi016 ceph-mon[64807]: pgmap v189: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 0 B/s wr, 217 op/s 2024-04-14T15:36:11.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:11 smithi016 ceph-mon[64807]: Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-04-14T15:36:11.568 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:11 smithi016 ceph-mon[64807]: osdmap e142: 8 total, 7 up, 8 in 2024-04-14T15:36:11.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:36:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:36:11] "GET /metrics HTTP/1.1" 200 38477 "" "Prometheus/2.33.4" 2024-04-14T15:36:11.902 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:11 smithi196 podman[68853]: 2024-04-14 15:36:11.885690565 +0000 UTC m=+0.544792587 container remove 8309d618722f1466788f74d9ebc5be57d0428a7a8f2a7a3c77487b72ba7b49a6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-04-14T15:36:12.165 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:11 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.7.service: Deactivated successfully. 2024-04-14T15:36:12.165 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:11 smithi196 systemd[1]: Stopped Ceph osd.7 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:36:12.165 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:11 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.7.service: Consumed 9.884s CPU time. 2024-04-14T15:36:12.165 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 systemd[1]: Starting Ceph osd.7 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:36:12.597 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 podman[68961]: 2024-04-14 15:36:12.221900204 +0000 UTC m=+0.018176588 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:36:12.597 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 podman[68961]: 2024-04-14 15:36:12.320110133 +0000 UTC m=+0.116386517 container create 085590546b007f7bfbef5a984689c9ee65c4daba02ebcdb91b897c3165a845c4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate, org.label-schema.build-date=20240403, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-04-14T15:36:12.597 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 podman[68961]: 2024-04-14 15:36:12.539383495 +0000 UTC m=+0.335659879 container init 085590546b007f7bfbef5a984689c9ee65c4daba02ebcdb91b897c3165a845c4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-14T15:36:12.597 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 podman[68961]: 2024-04-14 15:36:12.542598614 +0000 UTC m=+0.338875002 container start 085590546b007f7bfbef5a984689c9ee65c4daba02ebcdb91b897c3165a845c4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-14T15:36:12.597 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 podman[68961]: 2024-04-14 15:36:12.597446589 +0000 UTC m=+0.393722982 container attach 085590546b007f7bfbef5a984689c9ee65c4daba02ebcdb91b897c3165a845c4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-14T15:36:13.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:12 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:13.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:12 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:13.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:12 smithi196 ceph-mon[58679]: pgmap v191: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 0 B/s wr, 207 op/s 2024-04-14T15:36:13.165 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate[68976]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T15:36:13.165 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate[68976]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 bash[68961]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 bash[68961]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 bash[68961]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 bash[68961]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 bash[68961]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 bash[68961]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 bash[68961]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate[68976]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate[68976]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate[68976]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate[68976]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate[68976]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-14T15:36:13.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:13 smithi196 podman[68961]: 2024-04-14 15:36:13.005984581 +0000 UTC m=+0.802260968 container died 085590546b007f7bfbef5a984689c9ee65c4daba02ebcdb91b897c3165a845c4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:36:13.360 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:13.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:13.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:12 smithi016 ceph-mon[67348]: pgmap v191: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 0 B/s wr, 207 op/s 2024-04-14T15:36:13.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:13.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:13.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:12 smithi016 ceph-mon[64807]: pgmap v191: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 0 B/s wr, 207 op/s 2024-04-14T15:36:14.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:13 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:14.165 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:13 smithi196 podman[69092]: 2024-04-14 15:36:13.876517119 +0000 UTC m=+0.865069058 container remove 085590546b007f7bfbef5a984689c9ee65c4daba02ebcdb91b897c3165a845c4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-14T15:36:14.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:14 smithi196 podman[69159]: 2024-04-14 15:36:14.077681212 +0000 UTC m=+0.097165576 container create 1cf6aa633b3f94a239831a5ece70283908dc5062ce59c66eb7bd81924c040ee8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-14T15:36:14.166 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:14 smithi196 podman[69159]: 2024-04-14 15:36:13.998865028 +0000 UTC m=+0.018349391 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:36:14.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:13 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:14.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:13 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:14.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:14 smithi196 podman[69159]: 2024-04-14 15:36:14.296984858 +0000 UTC m=+0.316469231 container init 1cf6aa633b3f94a239831a5ece70283908dc5062ce59c66eb7bd81924c040ee8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:36:14.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:14 smithi196 podman[69159]: 2024-04-14 15:36:14.300218342 +0000 UTC m=+0.319702708 container start 1cf6aa633b3f94a239831a5ece70283908dc5062ce59c66eb7bd81924c040ee8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=) 2024-04-14T15:36:14.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:14 smithi196 ceph-osd[69178]: -- 172.21.15.196:0/1903225284 <== mon.2 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556c3b147680 con 0x556c3b111000 2024-04-14T15:36:14.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:14 smithi196 bash[69159]: 1cf6aa633b3f94a239831a5ece70283908dc5062ce59c66eb7bd81924c040ee8 2024-04-14T15:36:14.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:14 smithi196 systemd[1]: Started Ceph osd.7 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:36:15.165 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:14 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:36:14.839+0000 7efdb95977c0 -1 Falling back to public interface 2024-04-14T15:36:15.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:15 smithi196 ceph-mon[58679]: pgmap v192: 161 pgs: 9 active+undersized+degraded, 35 peering, 25 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 132 op/s; 24/627 objects degraded (3.828%) 2024-04-14T15:36:15.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:15 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:15.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:15 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:15.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:15 smithi196 ceph-mon[58679]: Health check failed: Degraded data redundancy: 24/627 objects degraded (3.828%), 9 pgs degraded (PG_DEGRADED) 2024-04-14T15:36:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:15 smithi016 ceph-mon[64807]: pgmap v192: 161 pgs: 9 active+undersized+degraded, 35 peering, 25 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 132 op/s; 24/627 objects degraded (3.828%) 2024-04-14T15:36:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:15 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:15 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:15 smithi016 ceph-mon[64807]: Health check failed: Degraded data redundancy: 24/627 objects degraded (3.828%), 9 pgs degraded (PG_DEGRADED) 2024-04-14T15:36:15.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:15 smithi016 ceph-mon[67348]: pgmap v192: 161 pgs: 9 active+undersized+degraded, 35 peering, 25 active+undersized, 92 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 132 op/s; 24/627 objects degraded (3.828%) 2024-04-14T15:36:15.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:15 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:15.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:15 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:15.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:15 smithi016 ceph-mon[67348]: Health check failed: Degraded data redundancy: 24/627 objects degraded (3.828%), 9 pgs degraded (PG_DEGRADED) 2024-04-14T15:36:16.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:16 smithi196 ceph-mon[58679]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-04-14T15:36:16.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:16 smithi016 ceph-mon[67348]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-04-14T15:36:16.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:16 smithi016 ceph-mon[64807]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-04-14T15:36:17.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:17 smithi196 ceph-mon[58679]: pgmap v193: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 98 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:17.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:17 smithi016 ceph-mon[67348]: pgmap v193: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 98 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:17.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:17 smithi016 ceph-mon[64807]: pgmap v193: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 98 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:17.900 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:17 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:36:17.590+0000 7efdb95977c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-04-14T15:36:17.900 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:17 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:36:17.621+0000 7efdb95977c0 -1 osd.7 140 log_to_monitors true 2024-04-14T15:36:18.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:18 smithi016 ceph-mon[67348]: from='osd.7 [v2:172.21.15.196:6824/2894977847,v1:172.21.15.196:6825/2894977847]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:36:18.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:18 smithi016 ceph-mon[67348]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:36:18.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:18 smithi016 ceph-mon[64807]: from='osd.7 [v2:172.21.15.196:6824/2894977847,v1:172.21.15.196:6825/2894977847]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:36:18.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:18 smithi016 ceph-mon[64807]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:36:18.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:18 smithi196 ceph-mon[58679]: from='osd.7 [v2:172.21.15.196:6824/2894977847,v1:172.21.15.196:6825/2894977847]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:36:18.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:18 smithi196 ceph-mon[58679]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T15:36:19.415 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:36:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:36:18.972+0000 7efda69e7700 -1 osd.7 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T15:36:19.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[67348]: pgmap v194: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:19.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[67348]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T15:36:19.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[67348]: osdmap e143: 8 total, 7 up, 8 in 2024-04-14T15:36:19.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[67348]: from='osd.7 [v2:172.21.15.196:6824/2894977847,v1:172.21.15.196:6825/2894977847]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:36:19.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[67348]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:36:19.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:19.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:19.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[67348]: from='osd.7 ' entity='osd.7' 2024-04-14T15:36:19.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[64807]: pgmap v194: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:19.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[64807]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T15:36:19.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[64807]: osdmap e143: 8 total, 7 up, 8 in 2024-04-14T15:36:19.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[64807]: from='osd.7 [v2:172.21.15.196:6824/2894977847,v1:172.21.15.196:6825/2894977847]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:36:19.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[64807]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:36:19.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:19.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:19.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:19 smithi016 ceph-mon[64807]: from='osd.7 ' entity='osd.7' 2024-04-14T15:36:19.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:19 smithi196 ceph-mon[58679]: pgmap v194: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:19 smithi196 ceph-mon[58679]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T15:36:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:19 smithi196 ceph-mon[58679]: osdmap e143: 8 total, 7 up, 8 in 2024-04-14T15:36:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:19 smithi196 ceph-mon[58679]: from='osd.7 [v2:172.21.15.196:6824/2894977847,v1:172.21.15.196:6825/2894977847]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:36:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:19 smithi196 ceph-mon[58679]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-04-14T15:36:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:19 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:19 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:19.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:19 smithi196 ceph-mon[58679]: from='osd.7 ' entity='osd.7' 2024-04-14T15:36:21.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[64807]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:36:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[64807]: osd.7 [v2:172.21.15.196:6824/2894977847,v1:172.21.15.196:6825/2894977847] boot 2024-04-14T15:36:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[64807]: osdmap e144: 8 total, 8 up, 8 in 2024-04-14T15:36:21.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:36:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[67348]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:36:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[67348]: osd.7 [v2:172.21.15.196:6824/2894977847,v1:172.21.15.196:6825/2894977847] boot 2024-04-14T15:36:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[67348]: osdmap e144: 8 total, 8 up, 8 in 2024-04-14T15:36:21.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:20 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:36:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:20 smithi196 ceph-mon[58679]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T15:36:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:20 smithi196 ceph-mon[58679]: osd.7 [v2:172.21.15.196:6824/2894977847,v1:172.21.15.196:6825/2894977847] boot 2024-04-14T15:36:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:20 smithi196 ceph-mon[58679]: osdmap e144: 8 total, 8 up, 8 in 2024-04-14T15:36:21.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:20 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T15:36:21.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:36:21 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:36:21] "GET /metrics HTTP/1.1" 200 38618 "" "Prometheus/2.33.4" 2024-04-14T15:36:21.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:21 smithi016 ceph-mon[67348]: pgmap v197: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 81/627 objects degraded (12.919%) 2024-04-14T15:36:21.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:21 smithi016 ceph-mon[67348]: osdmap e145: 8 total, 8 up, 8 in 2024-04-14T15:36:21.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:21 smithi016 ceph-mon[67348]: Health check update: Degraded data redundancy: 81/627 objects degraded (12.919%), 26 pgs degraded (PG_DEGRADED) 2024-04-14T15:36:21.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:21 smithi016 ceph-mon[64807]: pgmap v197: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 81/627 objects degraded (12.919%) 2024-04-14T15:36:21.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:21 smithi016 ceph-mon[64807]: osdmap e145: 8 total, 8 up, 8 in 2024-04-14T15:36:21.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:21 smithi016 ceph-mon[64807]: Health check update: Degraded data redundancy: 81/627 objects degraded (12.919%), 26 pgs degraded (PG_DEGRADED) 2024-04-14T15:36:21.903 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:21 smithi196 ceph-mon[58679]: pgmap v197: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 81/627 objects degraded (12.919%) 2024-04-14T15:36:21.904 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:21 smithi196 ceph-mon[58679]: osdmap e145: 8 total, 8 up, 8 in 2024-04-14T15:36:21.904 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:21 smithi196 ceph-mon[58679]: Health check update: Degraded data redundancy: 81/627 objects degraded (12.919%), 26 pgs degraded (PG_DEGRADED) 2024-04-14T15:36:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:36:22.723 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:36:22.723 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (6m) 2m ago 11m 23.9M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:36:22.723 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 error 4s ago 10m - - 2024-04-14T15:36:22.723 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (5m) 2m ago 10m 52.3M - 3.5 e1d6a67b021e 27d95fa4403f 2024-04-14T15:36:22.723 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (5m) 4s ago 15m 452M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:36:22.723 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (5m) 2m ago 17m 508M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:36:22.723 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (4m) 2m ago 17m 48.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:36:22.723 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (4m) 4s ago 15m 41.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 72817f80f666 2024-04-14T15:36:22.723 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (4m) 2m ago 16m 39.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (6m) 2m ago 11m 17.4M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (6m) 4s ago 11m 17.9M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (3m) 2m ago 15m 44.0M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 f6ee7262903f 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (2m) 2m ago 14m 43.7M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 14dec0c76aaf 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (2m) 2m ago 14m 51.5M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 ce205c8399d3 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (2m) 2m ago 14m 12.5M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 6925155fa41a 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (112s) 4s ago 13m 49.5M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 3a7864fa8601 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (77s) 4s ago 13m 48.3M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 08ddd21e0929 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (43s) 4s ago 13m 47.7M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 31fa3bea2859 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (8s) 4s ago 12m 12.6M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 1cf6aa633b3f 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (5m) 4s ago 11m 75.6M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (10m) 2m ago 10m 75.3M - 17.2.0 e1d6a67b021e 4701a7ad2ac4 2024-04-14T15:36:22.724 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (10m) 4s ago 10m 75.9M - 17.2.0 e1d6a67b021e a992a53749b7 2024-04-14T15:36:22.725 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (10m) 2m ago 10m 73.2M - 17.2.0 e1d6a67b021e 2faaf9e4141c 2024-04-14T15:36:22.725 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (10m) 4s ago 10m 76.5M - 17.2.0 e1d6a67b021e 20325379726b 2024-04-14T15:36:23.133 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-14T15:36:23.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:36:23.135 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:36:23.135 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-14T15:36:23.135 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 13 2024-04-14T15:36:23.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:36:23.135 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:36:23.500 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:36:23.500 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:36:23.501 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:36:23.501 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:36:23.501 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:36:23.501 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-04-14T15:36:23.501 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-14T15:36:23.501 INFO:teuthology.orchestra.run.smithi016.stdout: "osd" 2024-04-14T15:36:23.501 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:36:23.501 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "13/23 daemons upgraded", 2024-04-14T15:36:23.501 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T15:36:23.501 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:36:23.501 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:36:23.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='client.54418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:23.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: pgmap v199: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:23.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='client.54424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:23.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='client.44466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:23.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/2786046950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:23.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-14T15:36:23.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-14T15:36:23.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-14T15:36:23.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-14T15:36:23.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-14T15:36:23.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-14T15:36:23.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-14T15:36:23.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-14T15:36:23.863 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-14T15:36:23.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='client.54418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:23.863 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: pgmap v199: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='client.54424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='client.44466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/2786046950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:23.864 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:23.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-14T15:36:23.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-14T15:36:23.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-14T15:36:23.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-14T15:36:23.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-14T15:36:23.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-14T15:36:23.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-14T15:36:23.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-14T15:36:23.865 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-14T15:36:23.865 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-14T15:36:23.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-14T15:36:23.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-14T15:36:23.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-14T15:36:23.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-14T15:36:23.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-14T15:36:23.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-14T15:36:23.866 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-14T15:36:23.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-14T15:36:23.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-14T15:36:23.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-14T15:36:23.866 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-14T15:36:23.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-14T15:36:23.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-14T15:36:23.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-14T15:36:23.867 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:23 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-14T15:36:23.927 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 failed cephadm daemon(s); all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 81/627 objects degraded (12.919%), 26 pgs degraded 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in error state 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 81/627 objects degraded (12.919%), 26 pgs degraded 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-04-14T15:36:23.928 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-04-14T15:36:23.929 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-04-14T15:36:23.930 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-04-14T15:36:23.930 INFO:teuthology.orchestra.run.smithi016.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-04-14T15:36:23.930 INFO:teuthology.orchestra.run.smithi016.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-04-14T15:36:23.930 INFO:teuthology.orchestra.run.smithi016.stdout: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-04-14T15:36:23.930 INFO:teuthology.orchestra.run.smithi016.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-04-14T15:36:23.930 INFO:teuthology.orchestra.run.smithi016.stdout: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-04-14T15:36:24.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='client.54418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:24.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: pgmap v199: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:24.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='client.54424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:24.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='client.44466 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/2786046950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:24.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-14T15:36:24.167 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-14T15:36:24.168 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:23 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-14T15:36:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: pgmap v200: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: pgmap v201: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: Upgrade: Setting container_image for all osd 2024-04-14T15:36:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: from='client.54442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: Upgrade: Setting require_osd_release to 19 squid 2024-04-14T15:36:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/2404637066' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:36:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-14T15:36:25.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-14T15:36:25.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: osdmap e146: 8 total, 8 up, 8 in 2024-04-14T15:36:25.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:25.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: pgmap v200: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: pgmap v201: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: Upgrade: Setting container_image for all osd 2024-04-14T15:36:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: from='client.54442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: Upgrade: Setting require_osd_release to 19 squid 2024-04-14T15:36:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/2404637066' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:36:25.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-14T15:36:25.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-14T15:36:25.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: osdmap e146: 8 total, 8 up, 8 in 2024-04-14T15:36:25.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:25.113 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:24 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:25.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:25.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: pgmap v200: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:25.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: pgmap v201: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 81/627 objects degraded (12.919%) 2024-04-14T15:36:25.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: Upgrade: Setting container_image for all osd 2024-04-14T15:36:25.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: from='client.54442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:25.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: Upgrade: Setting require_osd_release to 19 squid 2024-04-14T15:36:25.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/2404637066' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:36:25.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-14T15:36:25.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-14T15:36:25.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: osdmap e146: 8 total, 8 up, 8 in 2024-04-14T15:36:25.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:25.166 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:24 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:26.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:25 smithi016 ceph-mon[67348]: Upgrade: Setting container_image for all mds 2024-04-14T15:36:26.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:25 smithi016 ceph-mon[64807]: Upgrade: Setting container_image for all mds 2024-04-14T15:36:26.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:25 smithi196 ceph-mon[58679]: Upgrade: Setting container_image for all mds 2024-04-14T15:36:27.078 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:26 smithi016 ceph-mon[67348]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-04-14T15:36:27.078 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:26 smithi016 ceph-mon[67348]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/627 objects degraded (12.919%), 26 pgs degraded) 2024-04-14T15:36:27.078 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:26 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:27.078 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:26 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:27.078 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:26 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:27.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:26 smithi016 ceph-mon[64807]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-04-14T15:36:27.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:26 smithi016 ceph-mon[64807]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/627 objects degraded (12.919%), 26 pgs degraded) 2024-04-14T15:36:27.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:26 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:27.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:26 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:27.079 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:26 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:26 smithi196 ceph-mon[58679]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-04-14T15:36:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:26 smithi196 ceph-mon[58679]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/627 objects degraded (12.919%), 26 pgs degraded) 2024-04-14T15:36:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:26 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:26 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi016.qozpor", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:27.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:26 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:28.065 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:27 smithi016 ceph-mon[67348]: Upgrade: Updating rgw.foo.smithi016.qozpor (1/4) 2024-04-14T15:36:28.066 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:27 smithi016 ceph-mon[67348]: Deploying daemon rgw.foo.smithi016.qozpor on smithi016 2024-04-14T15:36:28.066 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:28.066 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:28.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:27 smithi016 ceph-mon[64807]: Upgrade: Updating rgw.foo.smithi016.qozpor (1/4) 2024-04-14T15:36:28.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:27 smithi016 ceph-mon[64807]: Deploying daemon rgw.foo.smithi016.qozpor on smithi016 2024-04-14T15:36:28.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:28.066 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:27 smithi196 ceph-mon[58679]: Upgrade: Updating rgw.foo.smithi016.qozpor (1/4) 2024-04-14T15:36:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:27 smithi196 ceph-mon[58679]: Deploying daemon rgw.foo.smithi016.qozpor on smithi016 2024-04-14T15:36:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:28.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:28.852 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:28 smithi016 ceph-mon[67348]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-04-14T15:36:28.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:28 smithi016 ceph-mon[64807]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-04-14T15:36:29.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:28 smithi196 ceph-mon[58679]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-04-14T15:36:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:30 smithi196 ceph-mon[58679]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-04-14T15:36:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:30 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:30 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:31.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:30 smithi016 ceph-mon[67348]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-04-14T15:36:31.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:30 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:31.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:30 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:31.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:30 smithi016 ceph-mon[64807]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-04-14T15:36:31.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:30 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:31.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:30 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:31.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:36:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:36:31] "GET /metrics HTTP/1.1" 200 38630 "" "Prometheus/2.33.4" 2024-04-14T15:36:32.976 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:32 smithi016 ceph-mon[67348]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:36:32.976 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:32 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:32.976 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:32 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:32.976 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:32 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:32.976 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:32 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:32.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:32 smithi016 ceph-mon[64807]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:36:32.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:32 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:32.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:32 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:32.977 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:32 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:32.977 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:32 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:32 smithi196 ceph-mon[58679]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:36:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:32 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:32 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:32 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi016.sjsxjs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:33.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:32 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:33.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:33 smithi016 ceph-mon[67348]: Upgrade: Updating rgw.smpl.smithi016.sjsxjs (2/4) 2024-04-14T15:36:33.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:33 smithi016 ceph-mon[67348]: Deploying daemon rgw.smpl.smithi016.sjsxjs on smithi016 2024-04-14T15:36:33.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:33 smithi016 ceph-mon[64807]: Upgrade: Updating rgw.smpl.smithi016.sjsxjs (2/4) 2024-04-14T15:36:33.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:33 smithi016 ceph-mon[64807]: Deploying daemon rgw.smpl.smithi016.sjsxjs on smithi016 2024-04-14T15:36:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:33 smithi196 ceph-mon[58679]: Upgrade: Updating rgw.smpl.smithi016.sjsxjs (2/4) 2024-04-14T15:36:34.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:33 smithi196 ceph-mon[58679]: Deploying daemon rgw.smpl.smithi016.sjsxjs on smithi016 2024-04-14T15:36:34.982 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:34 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:34.982 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:34 smithi016 ceph-mon[67348]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:36:34.983 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:34 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:34.983 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:34 smithi016 ceph-mon[64807]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:36:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:34 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:35.165 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:34 smithi196 ceph-mon[58679]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:36:37.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:36 smithi196 ceph-mon[58679]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 94 B/s wr, 94 op/s 2024-04-14T15:36:37.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:36 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:37.091 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:36 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:37.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:36 smithi016 ceph-mon[67348]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 94 B/s wr, 94 op/s 2024-04-14T15:36:37.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:36 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:37.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:36 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:37.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:36 smithi016 ceph-mon[64807]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 94 B/s wr, 94 op/s 2024-04-14T15:36:37.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:36 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:37.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:36 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:38.908 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:38 smithi196 ceph-mon[58679]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-14T15:36:38.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:38 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:38.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:38 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:38.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:38 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:39.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:38 smithi016 ceph-mon[67348]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-14T15:36:39.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:38 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:39.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:38 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:39.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:38 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:39.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:38 smithi016 ceph-mon[64807]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-14T15:36:39.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:38 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:39.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:38 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi196.wjbrhf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:39.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:38 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:40.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:39 smithi196 ceph-mon[58679]: Upgrade: Updating rgw.foo.smithi196.wjbrhf (3/4) 2024-04-14T15:36:40.046 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:39 smithi196 ceph-mon[58679]: Deploying daemon rgw.foo.smithi196.wjbrhf on smithi196 2024-04-14T15:36:40.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:39 smithi016 ceph-mon[67348]: Upgrade: Updating rgw.foo.smithi196.wjbrhf (3/4) 2024-04-14T15:36:40.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:39 smithi016 ceph-mon[67348]: Deploying daemon rgw.foo.smithi196.wjbrhf on smithi196 2024-04-14T15:36:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:39 smithi016 ceph-mon[64807]: Upgrade: Updating rgw.foo.smithi196.wjbrhf (3/4) 2024-04-14T15:36:40.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:39 smithi016 ceph-mon[64807]: Deploying daemon rgw.foo.smithi196.wjbrhf on smithi196 2024-04-14T15:36:40.842 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:40 smithi196 ceph-mon[58679]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 85 B/s wr, 130 op/s 2024-04-14T15:36:41.110 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:40 smithi016 ceph-mon[64807]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 85 B/s wr, 130 op/s 2024-04-14T15:36:41.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:40 smithi016 ceph-mon[67348]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 85 B/s wr, 130 op/s 2024-04-14T15:36:41.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:36:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:36:41] "GET /metrics HTTP/1.1" 200 38630 "" "Prometheus/2.33.4" 2024-04-14T15:36:42.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:42 smithi016 ceph-mon[67348]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:42.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:42.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:42.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:42 smithi016 ceph-mon[64807]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:42.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:42.999 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:42 smithi196 ceph-mon[58679]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:42.999 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:42.999 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:43.000 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:44 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:44 smithi196 ceph-mon[58679]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:44 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:44 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:44 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:45.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:44 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:45.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:44 smithi016 ceph-mon[67348]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:45.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:44 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:45.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:44 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:45.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:44 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:45.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:44 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:45.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:44 smithi016 ceph-mon[64807]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:45.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:44 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:45.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:44 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi196.uwpszz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T15:36:45.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:44 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:45.812 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:45 smithi196 ceph-mon[58679]: Upgrade: Updating rgw.smpl.smithi196.uwpszz (4/4) 2024-04-14T15:36:45.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:45 smithi196 ceph-mon[58679]: Deploying daemon rgw.smpl.smithi196.uwpszz on smithi196 2024-04-14T15:36:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:45 smithi016 ceph-mon[64807]: Upgrade: Updating rgw.smpl.smithi196.uwpszz (4/4) 2024-04-14T15:36:46.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:45 smithi016 ceph-mon[64807]: Deploying daemon rgw.smpl.smithi196.uwpszz on smithi196 2024-04-14T15:36:46.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:45 smithi016 ceph-mon[67348]: Upgrade: Updating rgw.smpl.smithi196.uwpszz (4/4) 2024-04-14T15:36:46.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:45 smithi016 ceph-mon[67348]: Deploying daemon rgw.smpl.smithi196.uwpszz on smithi196 2024-04-14T15:36:46.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:46 smithi196 ceph-mon[58679]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-04-14T15:36:47.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:46 smithi016 ceph-mon[64807]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-04-14T15:36:47.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:46 smithi016 ceph-mon[67348]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 252 op/s 2024-04-14T15:36:49.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:48 smithi196 ceph-mon[58679]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:49.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:49.009 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:48 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:49.044 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:48 smithi016 ceph-mon[67348]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:49.044 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:49.044 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:48 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:49.045 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:48 smithi016 ceph-mon[64807]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:49.045 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:49.045 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:48 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:51.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:50 smithi016 ceph-mon[67348]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 170 B/s wr, 214 op/s 2024-04-14T15:36:51.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:50 smithi016 ceph-mon[64807]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 170 B/s wr, 214 op/s 2024-04-14T15:36:51.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:50 smithi196 ceph-mon[58679]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 170 B/s wr, 214 op/s 2024-04-14T15:36:51.860 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:36:51 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:36:51] "GET /metrics HTTP/1.1" 200 38631 "" "Prometheus/2.33.4" 2024-04-14T15:36:52.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:52 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:52.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:52 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:52.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:52 smithi016 ceph-mon[64807]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 255 B/s wr, 206 op/s 2024-04-14T15:36:52.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:52 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:52.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:52 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:52 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:52 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:52 smithi016 ceph-mon[67348]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 255 B/s wr, 206 op/s 2024-04-14T15:36:52.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:52 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:52.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:52 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:52 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:52 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:52 smithi196 ceph-mon[58679]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 255 B/s wr, 206 op/s 2024-04-14T15:36:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:52 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:52.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:52 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:53 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:53 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:53 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:53 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:53 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:53 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:53 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:53 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:53 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:53 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:53 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:53.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:53 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:54.300 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:36:54.464 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:54 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:54.465 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:54 smithi196 ceph-mon[58679]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:54.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:54 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:54.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:54 smithi016 ceph-mon[67348]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:54 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:36:54.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:54 smithi016 ceph-mon[64807]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:55.005 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:36:55.005 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (6m) 3s ago 11m 24.1M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:36:55.005 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 error 3s ago 11m - - 2024-04-14T15:36:55.005 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 running (5m) 3s ago 10m 58.0M - 3.5 e1d6a67b021e 27d95fa4403f 2024-04-14T15:36:55.005 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (5m) 3s ago 16m 452M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:36:55.005 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (6m) 3s ago 17m 515M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:36:55.005 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (5m) 3s ago 17m 53.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:36:55.005 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (4m) 3s ago 16m 42.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 72817f80f666 2024-04-14T15:36:55.005 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (4m) 3s ago 16m 44.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (7m) 3s ago 12m 17.6M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (6m) 3s ago 12m 18.2M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (3m) 3s ago 15m 54.9M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 f6ee7262903f 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (3m) 3s ago 15m 56.4M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 14dec0c76aaf 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (3m) 3s ago 15m 61.0M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 ce205c8399d3 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (2m) 3s ago 14m 63.0M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 6925155fa41a 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (2m) 3s ago 14m 54.5M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 3a7864fa8601 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (110s) 3s ago 14m 53.2M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 08ddd21e0929 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (75s) 3s ago 13m 52.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 31fa3bea2859 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (40s) 3s ago 13m 57.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 1cf6aa633b3f 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (5m) 3s ago 12m 75.7M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (25s) 3s ago 10m 75.2M - 19.0.0-2778-ga971fd5f b96c509eef84 733695267e90 2024-04-14T15:36:55.006 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (13s) 3s ago 10m 74.7M - 19.0.0-2778-ga971fd5f b96c509eef84 5b377dd47794 2024-04-14T15:36:55.007 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (19s) 3s ago 10m 75.2M - 19.0.0-2778-ga971fd5f b96c509eef84 3374290b1067 2024-04-14T15:36:55.007 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (7s) 3s ago 10m 77.2M - 19.0.0-2778-ga971fd5f b96c509eef84 2f5559477988 2024-04-14T15:36:55.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:55 smithi196 ceph-mon[58679]: from='client.54538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:55.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:55 smithi196 ceph-mon[58679]: from='client.44580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:55.433 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:36:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:36:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:36:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:36:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:36:55.433 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:36:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:36:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:36:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T15:36:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:36:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:36:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 4 2024-04-14T15:36:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:36:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:36:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 17 2024-04-14T15:36:55.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:36:55.434 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:36:55.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:55 smithi016 ceph-mon[67348]: from='client.54538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:55.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:55 smithi016 ceph-mon[67348]: from='client.44580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:55 smithi016 ceph-mon[64807]: from='client.54538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:55.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:55 smithi016 ceph-mon[64807]: from='client.44580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:55.783 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:36:55.783 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:36:55.783 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:36:55.783 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:36:55.783 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:36:55.783 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-04-14T15:36:55.784 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw", 2024-04-14T15:36:55.784 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-14T15:36:55.784 INFO:teuthology.orchestra.run.smithi016.stdout: "osd" 2024-04-14T15:36:55.784 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:36:55.784 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "17/23 daemons upgraded", 2024-04-14T15:36:55.784 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Currently upgrading rgw daemons", 2024-04-14T15:36:55.784 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": false 2024-04-14T15:36:55.784 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:36:56.249 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T15:36:56.249 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T15:36:56.249 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in error state 2024-04-14T15:36:56.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[67348]: from='client.54550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:56.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[67348]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:56.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/1281322686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:56.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:56.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:56.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/117291695' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:36:56.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[64807]: from='client.54550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:56.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[64807]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:56.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/1281322686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:56.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:56.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:56.612 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:56 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/117291695' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:36:56.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:56 smithi196 ceph-mon[58679]: from='client.54550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:56.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:56 smithi196 ceph-mon[58679]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T15:36:56.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:56 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/1281322686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:56.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:56 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:56.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:56 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:56.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:56 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/117291695' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:36:57.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='client.44598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:57.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: Detected new or changed devices on smithi196 2024-04-14T15:36:57.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.qozpor"}]: dispatch 2024-04-14T15:36:57.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.qozpor"}]': finished 2024-04-14T15:36:57.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wjbrhf"}]: dispatch 2024-04-14T15:36:57.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wjbrhf"}]': finished 2024-04-14T15:36:57.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.sjsxjs"}]: dispatch 2024-04-14T15:36:57.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.sjsxjs"}]': finished 2024-04-14T15:36:57.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.uwpszz"}]: dispatch 2024-04-14T15:36:57.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.uwpszz"}]': finished 2024-04-14T15:36:57.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.613 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.613 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='client.44598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: Detected new or changed devices on smithi196 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.614 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.qozpor"}]: dispatch 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.qozpor"}]': finished 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wjbrhf"}]: dispatch 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wjbrhf"}]': finished 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.sjsxjs"}]: dispatch 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.sjsxjs"}]': finished 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.uwpszz"}]: dispatch 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.uwpszz"}]': finished 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.615 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.616 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.616 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.616 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.616 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='client.44598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: Detected new or changed devices on smithi196 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.qozpor"}]: dispatch 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi016.qozpor"}]': finished 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wjbrhf"}]: dispatch 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi196.wjbrhf"}]': finished 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.sjsxjs"}]: dispatch 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi016.sjsxjs"}]': finished 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.uwpszz"}]: dispatch 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi196.uwpszz"}]': finished 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.667 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:57.668 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.668 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:36:57.668 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:58.610 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[64807]: Detected new or changed devices on smithi016 2024-04-14T15:36:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[64807]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 89 B/s wr, 88 op/s 2024-04-14T15:36:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[64807]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 107 B/s wr, 107 op/s 2024-04-14T15:36:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[64807]: Upgrade: Setting container_image for all rgw 2024-04-14T15:36:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[64807]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T15:36:58.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[64807]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T15:36:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[67348]: Detected new or changed devices on smithi016 2024-04-14T15:36:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[67348]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 89 B/s wr, 88 op/s 2024-04-14T15:36:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[67348]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 107 B/s wr, 107 op/s 2024-04-14T15:36:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[67348]: Upgrade: Setting container_image for all rgw 2024-04-14T15:36:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[67348]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T15:36:58.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:58 smithi016 ceph-mon[67348]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T15:36:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:58 smithi196 ceph-mon[58679]: Detected new or changed devices on smithi016 2024-04-14T15:36:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:58 smithi196 ceph-mon[58679]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 89 B/s wr, 88 op/s 2024-04-14T15:36:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:58 smithi196 ceph-mon[58679]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 107 B/s wr, 107 op/s 2024-04-14T15:36:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:58 smithi196 ceph-mon[58679]: Upgrade: Setting container_image for all rgw 2024-04-14T15:36:58.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:58 smithi196 ceph-mon[58679]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T15:36:58.666 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:58 smithi196 ceph-mon[58679]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T15:36:59.794 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:59 smithi016 ceph-mon[67348]: Upgrade: Updating iscsi.foo.smithi016.sijcrc 2024-04-14T15:36:59.794 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:59 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:59.794 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:59 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:36:59.795 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:59 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:59.795 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:36:59 smithi016 ceph-mon[67348]: Deploying daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:36:59.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:59 smithi016 ceph-mon[64807]: Upgrade: Updating iscsi.foo.smithi016.sijcrc 2024-04-14T15:36:59.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:59 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:59.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:59 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:36:59.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:59 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:59.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:36:59 smithi016 ceph-mon[64807]: Deploying daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:36:59.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:59 smithi196 ceph-mon[58679]: Upgrade: Updating iscsi.foo.smithi016.sijcrc 2024-04-14T15:36:59.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:59 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:36:59.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:59 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi016.sijcrc", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T15:36:59.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:59 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:36:59.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:36:59 smithi196 ceph-mon[58679]: Deploying daemon iscsi.foo.smithi016.sijcrc on smithi016 2024-04-14T15:37:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:00 smithi016 ceph-mon[64807]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 107 B/s wr, 48 op/s 2024-04-14T15:37:00.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:00 smithi016 ceph-mon[67348]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 107 B/s wr, 48 op/s 2024-04-14T15:37:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:00 smithi196 ceph-mon[58679]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 107 B/s wr, 48 op/s 2024-04-14T15:37:01.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:37:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:37:01] "GET /metrics HTTP/1.1" 200 38695 "" "Prometheus/2.33.4" 2024-04-14T15:37:02.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:02 smithi016 ceph-mon[64807]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:37:02.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:02 smithi016 ceph-mon[67348]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:37:02.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:02 smithi196 ceph-mon[58679]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:37:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:04 smithi016 ceph-mon[64807]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:37:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:04 smithi016 ceph-mon[64807]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:04 smithi016 ceph-mon[67348]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:37:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:04 smithi016 ceph-mon[67348]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:04 smithi196 ceph-mon[58679]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:37:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:04 smithi196 ceph-mon[58679]: from='client.25288 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:06.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:06 smithi016 ceph-mon[64807]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 539 B/s rd, 0 op/s 2024-04-14T15:37:06.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:06 smithi016 ceph-mon[67348]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 539 B/s rd, 0 op/s 2024-04-14T15:37:06.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:06 smithi196 ceph-mon[58679]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 539 B/s rd, 0 op/s 2024-04-14T15:37:08.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:08 smithi016 ceph-mon[64807]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:37:08.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:08 smithi016 ceph-mon[67348]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:37:08.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:08 smithi196 ceph-mon[58679]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T15:37:10.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:10 smithi016 ceph-mon[64807]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:10.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:10 smithi016 ceph-mon[67348]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:10.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:10 smithi196 ceph-mon[58679]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:11.610 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:37:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:37:11] "GET /metrics HTTP/1.1" 200 38695 "" "Prometheus/2.33.4" 2024-04-14T15:37:12.360 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:37:12 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:37:12.057+0000 7fda26792700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.sijcrc on host smithi016 failed. 2024-04-14T15:37:12.630 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:12 smithi016 ceph-mon[64807]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:12.630 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:12.630 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:12.631 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:12 smithi016 ceph-mon[67348]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:12.631 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:12.631 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:12.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:12 smithi196 ceph-mon[58679]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:12.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:12 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:12.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:12 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:13.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:13 smithi016 ceph-mon[67348]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.sijcrc on host smithi016 failed. 2024-04-14T15:37:13.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:13 smithi016 ceph-mon[67348]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-04-14T15:37:13.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:13 smithi016 ceph-mon[67348]: Health check failed: Upgrading daemon iscsi.foo.smithi016.sijcrc on host smithi016 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-14T15:37:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:13 smithi016 ceph-mon[64807]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.sijcrc on host smithi016 failed. 2024-04-14T15:37:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:13 smithi016 ceph-mon[64807]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-04-14T15:37:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:13 smithi016 ceph-mon[64807]: Health check failed: Upgrading daemon iscsi.foo.smithi016.sijcrc on host smithi016 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-14T15:37:13.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:13 smithi196 ceph-mon[58679]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.sijcrc on host smithi016 failed. 2024-04-14T15:37:13.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:13 smithi196 ceph-mon[58679]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-04-14T15:37:13.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:13 smithi196 ceph-mon[58679]: Health check failed: Upgrading daemon iscsi.foo.smithi016.sijcrc on host smithi016 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-14T15:37:15.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:15 smithi016 ceph-mon[67348]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-04-14T15:37:15.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:15 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:15.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:15 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:15 smithi016 ceph-mon[64807]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-04-14T15:37:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:15 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:15 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:15.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:15 smithi196 ceph-mon[58679]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-04-14T15:37:15.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:15 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:15.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:15 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:37:17.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:37:17.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-04-14T15:37:17.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-04-14T15:37:17.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: Checking dashboard <-> RGW credentials 2024-04-14T15:37:17.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:37:17.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:37:17.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-04-14T15:37:17.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:37:17.362 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: Checking dashboard <-> RGW credentials 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:37:17.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:16 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:37:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:37:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-04-14T15:37:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-04-14T15:37:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: Checking dashboard <-> RGW credentials 2024-04-14T15:37:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:17.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:37:17.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:37:17.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-04-14T15:37:17.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:37:17.416 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:16 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:18.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:17 smithi016 ceph-mon[67348]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:37:18.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:17 smithi016 ceph-mon[64807]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:37:18.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:17 smithi196 ceph-mon[58679]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T15:37:19.361 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:18 smithi016 ceph-mon[67348]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:37:19.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:18 smithi016 ceph-mon[64807]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:37:19.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:19 smithi196 ceph-mon[58679]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail 2024-04-14T15:37:21.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:21 smithi016 ceph-mon[64807]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-04-14T15:37:21.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:37:21 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:37:21] "GET /metrics HTTP/1.1" 200 38692 "" "Prometheus/2.33.4" 2024-04-14T15:37:21.862 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:21 smithi016 ceph-mon[67348]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-04-14T15:37:21.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:21 smithi196 ceph-mon[58679]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-04-14T15:37:23.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:23 smithi016 ceph-mon[64807]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-04-14T15:37:23.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:23 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/445009955' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:37:23.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:23 smithi016 ceph-mon[67348]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-04-14T15:37:23.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:23 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/445009955' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:37:23.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:23 smithi196 ceph-mon[58679]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-04-14T15:37:23.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:23 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/445009955' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T15:37:24.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:24 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/2692593198' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3070447653"}]: dispatch 2024-04-14T15:37:24.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:24 smithi016 ceph-mon[64807]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3070447653"}]: dispatch 2024-04-14T15:37:24.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:24 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/2692593198' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3070447653"}]: dispatch 2024-04-14T15:37:24.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:24 smithi016 ceph-mon[67348]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3070447653"}]: dispatch 2024-04-14T15:37:24.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:24 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/2692593198' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3070447653"}]: dispatch 2024-04-14T15:37:24.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:24 smithi196 ceph-mon[58679]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3070447653"}]: dispatch 2024-04-14T15:37:25.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:25 smithi016 ceph-mon[67348]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-04-14T15:37:25.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:25 smithi016 ceph-mon[67348]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3070447653"}]': finished 2024-04-14T15:37:25.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:25 smithi016 ceph-mon[67348]: osdmap e147: 8 total, 8 up, 8 in 2024-04-14T15:37:25.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:25 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/3314426129' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4070116660"}]: dispatch 2024-04-14T15:37:25.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:25 smithi016 ceph-mon[67348]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4070116660"}]: dispatch 2024-04-14T15:37:25.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:25 smithi016 ceph-mon[64807]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-04-14T15:37:25.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:25 smithi016 ceph-mon[64807]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3070447653"}]': finished 2024-04-14T15:37:25.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:25 smithi016 ceph-mon[64807]: osdmap e147: 8 total, 8 up, 8 in 2024-04-14T15:37:25.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:25 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/3314426129' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4070116660"}]: dispatch 2024-04-14T15:37:25.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:25 smithi016 ceph-mon[64807]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4070116660"}]: dispatch 2024-04-14T15:37:25.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:25 smithi196 ceph-mon[58679]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-04-14T15:37:25.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:25 smithi196 ceph-mon[58679]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/3070447653"}]': finished 2024-04-14T15:37:25.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:25 smithi196 ceph-mon[58679]: osdmap e147: 8 total, 8 up, 8 in 2024-04-14T15:37:25.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:25 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/3314426129' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4070116660"}]: dispatch 2024-04-14T15:37:25.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:25 smithi196 ceph-mon[58679]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4070116660"}]: dispatch 2024-04-14T15:37:26.629 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:37:26.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:26 smithi016 ceph-mon[67348]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4070116660"}]': finished 2024-04-14T15:37:26.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:26 smithi016 ceph-mon[67348]: osdmap e148: 8 total, 8 up, 8 in 2024-04-14T15:37:26.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:26 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/3193148130' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2849260632"}]: dispatch 2024-04-14T15:37:26.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:26 smithi016 ceph-mon[64807]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4070116660"}]': finished 2024-04-14T15:37:26.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:26 smithi016 ceph-mon[64807]: osdmap e148: 8 total, 8 up, 8 in 2024-04-14T15:37:26.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:26 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/3193148130' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2849260632"}]: dispatch 2024-04-14T15:37:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:26 smithi196 ceph-mon[58679]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/4070116660"}]': finished 2024-04-14T15:37:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:26 smithi196 ceph-mon[58679]: osdmap e148: 8 total, 8 up, 8 in 2024-04-14T15:37:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:26 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/3193148130' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2849260632"}]: dispatch 2024-04-14T15:37:26.980 INFO:teuthology.orchestra.run.smithi016.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.sijcrc on host smithi016 failed." 2024-04-14T15:37:27.599 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[67348]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-14T15:37:27.599 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/3193148130' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2849260632"}]': finished 2024-04-14T15:37:27.599 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[67348]: osdmap e149: 8 total, 8 up, 8 in 2024-04-14T15:37:27.599 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[67348]: from='client.44688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:37:27.599 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/1491755113' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2459572187"}]: dispatch 2024-04-14T15:37:27.599 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[67348]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2459572187"}]: dispatch 2024-04-14T15:37:27.599 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:27.599 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:27.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[64807]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-14T15:37:27.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/3193148130' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2849260632"}]': finished 2024-04-14T15:37:27.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[64807]: osdmap e149: 8 total, 8 up, 8 in 2024-04-14T15:37:27.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[64807]: from='client.44688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:37:27.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/1491755113' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2459572187"}]: dispatch 2024-04-14T15:37:27.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[64807]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2459572187"}]: dispatch 2024-04-14T15:37:27.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:27.601 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:27.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T15:37:27.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:27 smithi196 ceph-mon[58679]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-14T15:37:27.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:27 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/3193148130' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2849260632"}]': finished 2024-04-14T15:37:27.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:27 smithi196 ceph-mon[58679]: osdmap e149: 8 total, 8 up, 8 in 2024-04-14T15:37:27.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:27 smithi196 ceph-mon[58679]: from='client.44688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:37:27.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:27 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/1491755113' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2459572187"}]: dispatch 2024-04-14T15:37:27.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:27 smithi196 ceph-mon[58679]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2459572187"}]: dispatch 2024-04-14T15:37:27.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:37:27.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:28.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:28 smithi016 ceph-mon[67348]: from='client.54664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:37:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:28 smithi016 ceph-mon[67348]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2459572187"}]': finished 2024-04-14T15:37:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:28 smithi016 ceph-mon[67348]: osdmap e150: 8 total, 8 up, 8 in 2024-04-14T15:37:28.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:28 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/4049197731' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/630180669"}]: dispatch 2024-04-14T15:37:28.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:28 smithi016 ceph-mon[64807]: from='client.54664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:37:28.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:28 smithi016 ceph-mon[64807]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2459572187"}]': finished 2024-04-14T15:37:28.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:28 smithi016 ceph-mon[64807]: osdmap e150: 8 total, 8 up, 8 in 2024-04-14T15:37:28.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:28 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/4049197731' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/630180669"}]: dispatch 2024-04-14T15:37:28.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:28 smithi196 ceph-mon[58679]: from='client.54664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:37:28.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:28 smithi196 ceph-mon[58679]: from='client.? ' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:0/2459572187"}]': finished 2024-04-14T15:37:28.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:28 smithi196 ceph-mon[58679]: osdmap e150: 8 total, 8 up, 8 in 2024-04-14T15:37:28.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:28 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/4049197731' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/630180669"}]: dispatch 2024-04-14T15:37:29.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:29 smithi016 ceph-mon[64807]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:37:29.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:29 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/4049197731' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/630180669"}]': finished 2024-04-14T15:37:29.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:29 smithi016 ceph-mon[64807]: osdmap e151: 8 total, 8 up, 8 in 2024-04-14T15:37:29.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:29 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/939076100' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/630180669"}]: dispatch 2024-04-14T15:37:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:29 smithi016 ceph-mon[67348]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:37:29.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:29 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/4049197731' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/630180669"}]': finished 2024-04-14T15:37:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:29 smithi016 ceph-mon[67348]: osdmap e151: 8 total, 8 up, 8 in 2024-04-14T15:37:29.612 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:29 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/939076100' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/630180669"}]: dispatch 2024-04-14T15:37:29.751 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (7m) 13s ago 12m 24.1M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 error 14s ago 11m - - 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 unknown 13s ago 11m - - 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (6m) 14s ago 16m 452M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (6m) 13s ago 18m 516M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (5m) 13s ago 18m 55.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (5m) 14s ago 17m 43.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 72817f80f666 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (5m) 13s ago 17m 46.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (7m) 13s ago 12m 18.1M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (7m) 14s ago 12m 18.2M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (4m) 13s ago 16m 54.1M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 f6ee7262903f 2024-04-14T15:37:29.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (3m) 13s ago 16m 55.9M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 14dec0c76aaf 2024-04-14T15:37:29.753 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (3m) 13s ago 15m 61.1M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 ce205c8399d3 2024-04-14T15:37:29.753 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (3m) 13s ago 15m 63.2M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 6925155fa41a 2024-04-14T15:37:29.753 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (2m) 14s ago 14m 54.7M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 3a7864fa8601 2024-04-14T15:37:29.753 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (2m) 14s ago 14m 52.4M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 08ddd21e0929 2024-04-14T15:37:29.753 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (110s) 14s ago 14m 52.1M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 31fa3bea2859 2024-04-14T15:37:29.753 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (75s) 14s ago 13m 57.0M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 1cf6aa633b3f 2024-04-14T15:37:29.753 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (6m) 14s ago 13m 75.7M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:37:29.753 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (59s) 13s ago 11m 75.5M - 19.0.0-2778-ga971fd5f b96c509eef84 733695267e90 2024-04-14T15:37:29.753 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (48s) 14s ago 11m 75.0M - 19.0.0-2778-ga971fd5f b96c509eef84 5b377dd47794 2024-04-14T15:37:29.753 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (53s) 13s ago 11m 75.5M - 19.0.0-2778-ga971fd5f b96c509eef84 3374290b1067 2024-04-14T15:37:29.753 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (42s) 14s ago 11m 77.6M - 19.0.0-2778-ga971fd5f b96c509eef84 2f5559477988 2024-04-14T15:37:29.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:29 smithi196 ceph-mon[58679]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:37:29.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:29 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/4049197731' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6800/630180669"}]': finished 2024-04-14T15:37:29.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:29 smithi196 ceph-mon[58679]: osdmap e151: 8 total, 8 up, 8 in 2024-04-14T15:37:29.817 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:29 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/939076100' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/630180669"}]: dispatch 2024-04-14T15:37:30.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions' 2024-04-14T15:37:30.758 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:30 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/939076100' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/630180669"}]': finished 2024-04-14T15:37:30.758 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:30 smithi016 ceph-mon[67348]: osdmap e152: 8 total, 8 up, 8 in 2024-04-14T15:37:30.758 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:30 smithi016 ceph-mon[67348]: from='client.54682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:37:30.758 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:30 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/939076100' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/630180669"}]': finished 2024-04-14T15:37:30.758 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:30 smithi016 ceph-mon[64807]: osdmap e152: 8 total, 8 up, 8 in 2024-04-14T15:37:30.758 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:30 smithi016 ceph-mon[64807]: from='client.54682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:37:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:30 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/939076100' entity='client.iscsi.foo.smithi016.sijcrc' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.16:6801/630180669"}]': finished 2024-04-14T15:37:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:30 smithi196 ceph-mon[58679]: osdmap e152: 8 total, 8 up, 8 in 2024-04-14T15:37:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:30 smithi196 ceph-mon[58679]: from='client.54682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:37:31.699 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:31 smithi016 ceph-mon[67348]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:31.700 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:31 smithi016 ceph-mon[64807]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:31.700 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:37:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:37:31] "GET /metrics HTTP/1.1" 200 38773 "" "Prometheus/2.33.4" 2024-04-14T15:37:31.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:31 smithi196 ceph-mon[58679]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:32.520 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:37:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:37:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:37:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:37:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:37:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:37:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:37:32.520 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:37:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T15:37:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:37:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:37:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 4 2024-04-14T15:37:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:37:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:37:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 17 2024-04-14T15:37:32.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:37:32.521 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:37:33.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-14T15:37:33.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:33 smithi016 ceph-mon[67348]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-04-14T15:37:33.771 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:33 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/4116981218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:37:33.771 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:33 smithi016 ceph-mon[64807]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-04-14T15:37:33.772 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:33 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/4116981218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:37:33.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:33 smithi196 ceph-mon[58679]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-04-14T15:37:33.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:33 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/4116981218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:37:34.733 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:34 smithi016 ceph-mon[67348]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:34.734 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:34 smithi016 ceph-mon[64807]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:34 smithi196 ceph-mon[58679]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:35.038 INFO:teuthology.orchestra.run.smithi016.stdout:wait for servicemap items w/ changing names to refresh 2024-04-14T15:37:35.743 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:35 smithi016 ceph-mon[64807]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-04-14T15:37:35.744 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:35 smithi016 ceph-mon[67348]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-04-14T15:37:35.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'sleep 60' 2024-04-14T15:37:35.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:35 smithi196 ceph-mon[58679]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-04-14T15:37:37.611 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:37 smithi016 ceph-mon[67348]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:37.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:37 smithi016 ceph-mon[64807]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:37.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:37 smithi196 ceph-mon[58679]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:39.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:39 smithi016 ceph-mon[64807]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:37:39.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:39 smithi016 ceph-mon[67348]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:37:39.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:39 smithi196 ceph-mon[58679]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:37:41.860 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:41 smithi016 ceph-mon[64807]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:37:41.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:41 smithi016 ceph-mon[67348]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:37:41.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:37:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:37:41] "GET /metrics HTTP/1.1" 200 38773 "" "Prometheus/2.33.4" 2024-04-14T15:37:41.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:41 smithi196 ceph-mon[58679]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T15:37:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:43.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:43 smithi016 ceph-mon[67348]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:43.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:43 smithi016 ceph-mon[64807]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:43 smithi196 ceph-mon[58679]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:44.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:44 smithi016 ceph-mon[67348]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:44.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:44 smithi016 ceph-mon[64807]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:44 smithi196 ceph-mon[58679]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:45.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:45 smithi016 ceph-mon[67348]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:45.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:45 smithi016 ceph-mon[64807]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:45.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:45 smithi196 ceph-mon[58679]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:47.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:47 smithi016 ceph-mon[64807]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:47.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:47 smithi016 ceph-mon[67348]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:47.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:47 smithi196 ceph-mon[58679]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:49.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:49 smithi016 ceph-mon[67348]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:49.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:49 smithi016 ceph-mon[64807]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:49.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:49 smithi196 ceph-mon[58679]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:51.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:51 smithi016 ceph-mon[64807]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:51.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:51 smithi016 ceph-mon[67348]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:51.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:37:51 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:37:51] "GET /metrics HTTP/1.1" 200 38771 "" "Prometheus/2.33.4" 2024-04-14T15:37:51.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:51 smithi196 ceph-mon[58679]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:53.617 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:53 smithi016 ceph-mon[67348]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:53.617 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:53 smithi016 ceph-mon[64807]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:53.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:53 smithi196 ceph-mon[58679]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:54.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:54 smithi016 ceph-mon[67348]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:54.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:54 smithi016 ceph-mon[64807]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:54 smithi196 ceph-mon[58679]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:37:55.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:55 smithi016 ceph-mon[67348]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:55.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:55 smithi016 ceph-mon[64807]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:55.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:55 smithi196 ceph-mon[58679]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:57.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:57 smithi016 ceph-mon[67348]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:57.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:57.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:57 smithi016 ceph-mon[64807]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:57.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:57.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:57 smithi196 ceph-mon[58679]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:37:57.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:37:59.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:37:59 smithi016 ceph-mon[67348]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:59.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:37:59 smithi016 ceph-mon[64807]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:37:59.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:37:59 smithi196 ceph-mon[58679]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:01.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:01 smithi016 ceph-mon[67348]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:01.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:01 smithi016 ceph-mon[64807]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:01.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:38:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:38:01] "GET /metrics HTTP/1.1" 200 38768 "" "Prometheus/2.33.4" 2024-04-14T15:38:01.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:01 smithi196 ceph-mon[58679]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:03.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:03 smithi016 ceph-mon[67348]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:03.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:03 smithi016 ceph-mon[64807]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:03.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:03 smithi196 ceph-mon[58679]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:04 smithi016 ceph-mon[67348]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:04 smithi016 ceph-mon[64807]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:04 smithi196 ceph-mon[58679]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:05.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:05 smithi016 ceph-mon[67348]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:05.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:05 smithi016 ceph-mon[64807]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:05.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:05 smithi196 ceph-mon[58679]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:07.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:07 smithi016 ceph-mon[67348]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:07.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:07 smithi016 ceph-mon[64807]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:07.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:07 smithi196 ceph-mon[58679]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:09.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:09 smithi016 ceph-mon[67348]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:09.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:09 smithi016 ceph-mon[64807]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:09.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:09 smithi196 ceph-mon[58679]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:11.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:11 smithi016 ceph-mon[67348]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:11.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:11 smithi016 ceph-mon[64807]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:11.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:38:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:38:11] "GET /metrics HTTP/1.1" 200 38768 "" "Prometheus/2.33.4" 2024-04-14T15:38:11.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:11 smithi196 ceph-mon[58679]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:12.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:12 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:12.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:12 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:12.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:12 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:13.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:13 smithi016 ceph-mon[67348]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:13.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:13 smithi016 ceph-mon[64807]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:13.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:13 smithi196 ceph-mon[58679]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:14.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:14 smithi016 ceph-mon[67348]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:14.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:14 smithi016 ceph-mon[64807]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:14.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:14 smithi196 ceph-mon[58679]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:15.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:15 smithi016 ceph-mon[67348]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:15.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:15 smithi016 ceph-mon[64807]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:15.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:15 smithi196 ceph-mon[58679]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:17.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:17 smithi016 ceph-mon[67348]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:17.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:17 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:38:17.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:17 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:38:17.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:17 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:38:17.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:17 smithi016 ceph-mon[64807]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:17.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:17 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:38:17.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:17 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:38:17.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:17 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:38:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:17 smithi196 ceph-mon[58679]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:17 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T15:38:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:17 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T15:38:17.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:17 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' 2024-04-14T15:38:18.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:18 smithi016 ceph-mon[67348]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-04-14T15:38:18.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:18 smithi016 ceph-mon[67348]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:38:18.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:18 smithi016 ceph-mon[64807]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-04-14T15:38:18.862 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:18 smithi016 ceph-mon[64807]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:38:18.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:18 smithi196 ceph-mon[58679]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-04-14T15:38:18.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:18 smithi196 ceph-mon[58679]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T15:38:20.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:20 smithi016 ceph-mon[67348]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-04-14T15:38:20.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:20 smithi016 ceph-mon[64807]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-04-14T15:38:20.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:20 smithi196 ceph-mon[58679]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-04-14T15:38:21.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:38:21 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:38:21] "GET /metrics HTTP/1.1" 200 38770 "" "Prometheus/2.33.4" 2024-04-14T15:38:22.860 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:22 smithi016 ceph-mon[67348]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:38:22.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:22 smithi016 ceph-mon[64807]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:38:22.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:22 smithi196 ceph-mon[58679]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:38:24.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:24 smithi016 ceph-mon[67348]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:24.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:24 smithi016 ceph-mon[67348]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:38:24.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:24 smithi016 ceph-mon[64807]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:24.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:24 smithi016 ceph-mon[64807]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:38:24.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:24 smithi196 ceph-mon[58679]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:24.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:24 smithi196 ceph-mon[58679]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:38:26.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:26 smithi016 ceph-mon[67348]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:38:26.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:26 smithi016 ceph-mon[64807]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:38:26.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:26 smithi196 ceph-mon[58679]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T15:38:27.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:27 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:27.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:27 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:27.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:27 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:28.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:28 smithi016 ceph-mon[67348]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-14T15:38:28.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:28 smithi016 ceph-mon[64807]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-14T15:38:28.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:28 smithi196 ceph-mon[58679]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-14T15:38:30.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:30 smithi016 ceph-mon[67348]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:38:30.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:30 smithi016 ceph-mon[64807]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:38:30.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:30 smithi196 ceph-mon[58679]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T15:38:31.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:38:31 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:38:31] "GET /metrics HTTP/1.1" 200 38770 "" "Prometheus/2.33.4" 2024-04-14T15:38:32.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:32 smithi016 ceph-mon[67348]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:32.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:32 smithi016 ceph-mon[64807]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:32.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:32 smithi196 ceph-mon[58679]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:34.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:34 smithi016 ceph-mon[67348]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:34.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:34 smithi016 ceph-mon[67348]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:34.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:34 smithi016 ceph-mon[64807]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:34.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:34 smithi016 ceph-mon[64807]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:34 smithi196 ceph-mon[58679]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:34.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:34 smithi196 ceph-mon[58679]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:36.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:36 smithi016 ceph-mon[67348]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:36.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:36 smithi016 ceph-mon[64807]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:36.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:36 smithi196 ceph-mon[58679]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:38.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T15:38:38.779 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:38 smithi016 ceph-mon[67348]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:38.779 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:38 smithi016 ceph-mon[64807]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:38.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:38 smithi196 ceph-mon[58679]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T15:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:alertmanager.a smithi016 *:9093,9094 running (8m) 84s ago 13m 24.1M - 0.23.0 ba2b418f427c a29a63fa7146 2024-04-14T15:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:grafana.a smithi196 *:3000 error 84s ago 13m - - 2024-04-14T15:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:iscsi.foo.smithi016.sijcrc smithi016 unknown 84s ago 12m - - 2024-04-14T15:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.x smithi196 *:8443,9283,8765 running (7m) 84s ago 17m 452M - 19.0.0-2778-ga971fd5f b96c509eef84 c87a0109a594 2024-04-14T15:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:mgr.y smithi016 *:8443,9283,8765 running (8m) 84s ago 19m 516M - 19.0.0-2778-ga971fd5f b96c509eef84 e9559f366129 2024-04-14T15:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a smithi016 running (7m) 84s ago 19m 55.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 152c4b9fda20 2024-04-14T15:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b smithi196 running (6m) 84s ago 18m 43.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 72817f80f666 2024-04-14T15:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c smithi016 running (6m) 84s ago 18m 46.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 da68d9a2587c 2024-04-14T15:38:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.a smithi016 *:9100 running (8m) 84s ago 14m 18.1M - 1.3.1 1dbe0e931976 cb8e94cfa1ed 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:node-exporter.b smithi196 *:9100 running (8m) 84s ago 14m 18.2M - 1.3.1 1dbe0e931976 bea8d0b8e46e 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 smithi016 running (5m) 84s ago 17m 54.1M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 f6ee7262903f 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 smithi016 running (5m) 84s ago 17m 55.9M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 14dec0c76aaf 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 smithi016 running (4m) 84s ago 16m 61.1M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 ce205c8399d3 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 smithi016 running (4m) 84s ago 16m 63.2M 2737M 19.0.0-2778-ga971fd5f b96c509eef84 6925155fa41a 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 smithi196 running (4m) 84s ago 16m 54.7M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 3a7864fa8601 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 smithi196 running (3m) 84s ago 15m 52.4M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 08ddd21e0929 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:osd.6 smithi196 running (3m) 84s ago 15m 52.1M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 31fa3bea2859 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:osd.7 smithi196 running (2m) 84s ago 14m 57.0M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 1cf6aa633b3f 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:prometheus.a smithi196 *:9095 running (7m) 84s ago 14m 75.7M - 2.33.4 514e6a882f6e 9f60db35d8a8 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi016.qozpor smithi016 *:8000 running (2m) 84s ago 12m 75.5M - 19.0.0-2778-ga971fd5f b96c509eef84 733695267e90 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.foo.smithi196.wjbrhf smithi196 *:8000 running (118s) 84s ago 12m 75.0M - 19.0.0-2778-ga971fd5f b96c509eef84 5b377dd47794 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi016.sjsxjs smithi016 *:80 running (2m) 84s ago 12m 75.5M - 19.0.0-2778-ga971fd5f b96c509eef84 3374290b1067 2024-04-14T15:38:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:rgw.smpl.smithi196.uwpszz smithi196 *:80 running (112s) 84s ago 12m 77.6M - 19.0.0-2778-ga971fd5f b96c509eef84 2f5559477988 2024-04-14T15:38:40.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:40 smithi016 ceph-mon[64807]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:40.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:40 smithi016 ceph-mon[67348]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:40.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:40 smithi196 ceph-mon[58679]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:40.932 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions' 2024-04-14T15:38:41.665 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:41 smithi196 ceph-mon[58679]: from='client.44730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:38:41.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:41 smithi016 ceph-mon[67348]: from='client.44730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:38:41.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:41 smithi016 ceph-mon[64807]: from='client.44730 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:38:41.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:38:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:38:41] "GET /metrics HTTP/1.1" 200 38770 "" "Prometheus/2.33.4" 2024-04-14T15:38:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:42 smithi016 ceph-mon[64807]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:42.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:42 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:42 smithi016 ceph-mon[67348]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:42.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:42 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:42 smithi196 ceph-mon[58679]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:42.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:42 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:43.103 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:38:43.103 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": { 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": { 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": { 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw": { 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 4 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "overall": { 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 17 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-14T15:38:43.104 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:38:43.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:43 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/1772730570' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:38:43.759 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:43 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/1772730570' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:38:43.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade status' 2024-04-14T15:38:43.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:43 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/1772730570' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:38:44.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:44 smithi016 ceph-mon[67348]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:44.750 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:44 smithi016 ceph-mon[67348]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:44 smithi016 ceph-mon[64807]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:44.751 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:44 smithi016 ceph-mon[64807]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:44 smithi196 ceph-mon[58679]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:44.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:44 smithi196 ceph-mon[58679]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:45.755 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "in_progress": true, 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "services_complete": [ 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mon", 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw", 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr", 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "osd" 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "progress": "17/23 daemons upgraded", 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.sijcrc on host smithi016 failed.", 2024-04-14T15:38:45.756 INFO:teuthology.orchestra.run.smithi016.stdout: "is_paused": true 2024-04-14T15:38:45.757 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:38:46.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T15:38:46.858 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:46 smithi016 ceph-mon[67348]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:46.859 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:46 smithi016 ceph-mon[67348]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:38:46.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:46 smithi016 ceph-mon[64807]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:46.859 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:46 smithi016 ceph-mon[64807]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:38:46.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:46 smithi196 ceph-mon[58679]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:46.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:46 smithi196 ceph-mon[58679]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:38:48.535 INFO:teuthology.orchestra.run.smithi016.stdout:HEALTH_WARN 2 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi016.sijcrc on host smithi016 failed. 2024-04-14T15:38:48.535 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-14T15:38:48.535 INFO:teuthology.orchestra.run.smithi016.stdout: daemon iscsi.foo.smithi016.sijcrc on smithi016 is in unknown state 2024-04-14T15:38:48.535 INFO:teuthology.orchestra.run.smithi016.stdout: daemon grafana.a on smithi196 is in error state 2024-04-14T15:38:48.535 INFO:teuthology.orchestra.run.smithi016.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi016.sijcrc on host smithi016 failed. 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrade daemon: iscsi.foo.smithi016.sijcrc: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi016.sijcrc ... 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout:Creating ceph-iscsi config... 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout:Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/iscsi-gateway.cfg 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout:Write file: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/iscsi.foo.smithi016.sijcrc/tcmu-runner-entrypoint.sh 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d3ca5917a\x2dfa72\x2d11ee\x2dbc8e\x2dc7b262605968.slice/ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl start ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout:Traceback (most recent call last): 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: return _run_code(code, main_globals, None, 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T15:38:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: exec(code, run_globals) 2024-04-14T15:38:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T15:38:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T15:38:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T15:38:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T15:38:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T15:38:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1087, in deploy_daemon 2024-04-14T15:38:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1223, in deploy_daemon_units 2024-04-14T15:38:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: File "/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T15:38:48.537 INFO:teuthology.orchestra.run.smithi016.stdout:RuntimeError: Failed command: systemctl start ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc: Job for ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service failed because the control process exited with error code. 2024-04-14T15:38:48.537 INFO:teuthology.orchestra.run.smithi016.stdout:See "systemctl status ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" and "journalctl -xeu ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@iscsi.foo.smithi016.sijcrc.service" for details. 2024-04-14T15:38:48.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:48 smithi016 ceph-mon[64807]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:48.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:48 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/722499901' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:38:48.842 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:48 smithi016 ceph-mon[67348]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:48.842 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:48 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/722499901' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:38:48.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:48 smithi196 ceph-mon[58679]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:48.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:48 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/722499901' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T15:38:49.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-14T15:38:50.781 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:50 smithi016 ceph-mon[67348]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:50.781 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:50 smithi016 ceph-mon[64807]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:50.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:50 smithi196 ceph-mon[58679]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:51.350 INFO:teuthology.orchestra.run.smithi016.stdout:true 2024-04-14T15:38:51.611 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:38:51 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:38:51] "GET /metrics HTTP/1.1" 200 38770 "" "Prometheus/2.33.4" 2024-04-14T15:38:51.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:51 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/3867570738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:38:51.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:51 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/3867570738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:38:51.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:51 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/3867570738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:38:51.931 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-14T15:38:52.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:52 smithi016 ceph-mon[67348]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:52.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:52 smithi016 ceph-mon[64807]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:52.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:52 smithi196 ceph-mon[58679]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:54.023 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)" 2024-04-14T15:38:54.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-04-14T15:38:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:54 smithi196 ceph-mon[58679]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:54 smithi196 ceph-mon[58679]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:54.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:54 smithi196 ceph-mon[58679]: from='client.? 172.21.15.16:0/1643841469' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:38:54.974 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:54 smithi016 ceph-mon[67348]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:54.974 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:54 smithi016 ceph-mon[67348]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:54.974 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:54 smithi016 ceph-mon[67348]: from='client.? 172.21.15.16:0/1643841469' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:38:54.974 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:54 smithi016 ceph-mon[64807]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:38:54.974 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:54 smithi016 ceph-mon[64807]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:54.974 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:54 smithi016 ceph-mon[64807]: from='client.? 172.21.15.16:0/1643841469' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T15:38:56.669 INFO:teuthology.orchestra.run.smithi016.stdout:osd 8 101s ago - 2024-04-14T15:38:56.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:56 smithi196 ceph-mon[58679]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:57.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:56 smithi016 ceph-mon[67348]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:57.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:56 smithi016 ceph-mon[64807]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:38:57.434 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T15:38:57.445 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi016.front.sepia.ceph.com 2024-04-14T15:38:57.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-04-14T15:38:57.760 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:57 smithi016 ceph-mon[67348]: from='client.44766 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:38:57.760 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:57 smithi016 ceph-mon[67348]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:57.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:57 smithi016 ceph-mon[64807]: from='client.44766 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:38:57.760 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:57 smithi016 ceph-mon[64807]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:57.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:57 smithi196 ceph-mon[58679]: from='client.44766 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:38:57.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:57 smithi196 ceph-mon[58679]: from='mgr.25330 172.21.15.16:0/810069357' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T15:38:58.674 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:38:58 smithi016 ceph-mon[67348]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:58.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:38:58 smithi016 ceph-mon[64807]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:38:58.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:38:58 smithi196 ceph-mon[58679]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:39:00.225 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-14T15:39:00.225 INFO:teuthology.orchestra.run.smithi016.stdout: "image": "quay.io/ceph/ceph", 2024-04-14T15:39:00.225 INFO:teuthology.orchestra.run.smithi016.stdout: "registry": "quay.io", 2024-04-14T15:39:00.225 INFO:teuthology.orchestra.run.smithi016.stdout: "bare_image": "ceph/ceph", 2024-04-14T15:39:00.225 INFO:teuthology.orchestra.run.smithi016.stdout: "versions": [] 2024-04-14T15:39:00.225 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-14T15:39:00.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:00 smithi016 ceph-mon[67348]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:39:00.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:00 smithi016 ceph-mon[67348]: from='client.44772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:39:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:00 smithi016 ceph-mon[64807]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:39:00.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:00 smithi016 ceph-mon[64807]: from='client.44772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:39:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:00 smithi196 ceph-mon[58679]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:39:00.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:00 smithi196 ceph-mon[58679]: from='client.44772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:39:01.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-04-14T15:39:01.713 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:01 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:39:01] "GET /metrics HTTP/1.1" 200 38771 "" "Prometheus/2.33.4" 2024-04-14T15:39:02.802 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:02 smithi016 ceph-mon[67348]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:39:02.802 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:02 smithi016 ceph-mon[64807]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:39:02.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:02 smithi196 ceph-mon[58679]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:39:03.848 INFO:teuthology.orchestra.run.smithi016.stdout: "16.2.0", 2024-04-14T15:39:04.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-04-14T15:39:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:04 smithi016 ceph-mon[64807]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:39:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:04 smithi016 ceph-mon[64807]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:39:04.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:04 smithi016 ceph-mon[64807]: from='client.44778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:39:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:04 smithi016 ceph-mon[67348]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:39:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:04 smithi016 ceph-mon[67348]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:39:04.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:04 smithi016 ceph-mon[67348]: from='client.44778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:39:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:04 smithi196 ceph-mon[58679]: from='client.44658 -' entity='client.iscsi.foo.smithi016.sijcrc' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T15:39:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:04 smithi196 ceph-mon[58679]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T15:39:04.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:04 smithi196 ceph-mon[58679]: from='client.44778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:39:06.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:06 smithi196 ceph-mon[58679]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:39:07.110 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:06 smithi016 ceph-mon[67348]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:39:07.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:06 smithi016 ceph-mon[64807]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T15:39:07.284 INFO:teuthology.orchestra.run.smithi016.stdout: "v16.2.2", 2024-04-14T15:39:07.285 INFO:teuthology.orchestra.run.smithi016.stdout: "v16.2.2-20210505", 2024-04-14T15:39:07.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:07 smithi016 ceph-mon[64807]: from='client.54748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:39:07.861 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:07 smithi016 ceph-mon[67348]: from='client.54748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:39:07.996 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-14T15:39:08.006 INFO:tasks.cephadm:Teardown begin 2024-04-14T15:39:08.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:39:08.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:39:08.070 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:07 smithi196 ceph-mon[58679]: from='client.54748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T15:39:08.076 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-14T15:39:08.076 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-14T15:39:08.095 DEBUG:teuthology.orchestra.run.smithi196:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-14T15:39:08.133 INFO:tasks.cephadm:Stopping all daemons... 2024-04-14T15:39:08.134 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-14T15:39:08.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.a 2024-04-14T15:39:08.361 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:08 smithi016 systemd[1]: Stopping Ceph mon.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:39:08.840 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a[64783]: 2024-04-14T15:39:08.455+0000 7f94cfe8c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:39:08.840 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a[64783]: 2024-04-14T15:39:08.455+0000 7f94cfe8c700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-04-14T15:39:08.841 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:08 smithi016 podman[91224]: 2024-04-14 15:39:08.496537948 +0000 UTC m=+0.170596369 container died 152c4b9fda2092b8d247f1295f7f190fc009ae33dfb9121dca73fe144e2203f6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, RELEASE=main-a971fd5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-14T15:39:08.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:39:08] ENGINE Bus STOPPING 2024-04-14T15:39:08.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:39:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T15:39:08.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:39:08] ENGINE Bus STOPPED 2024-04-14T15:39:08.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:39:08] ENGINE Bus STARTING 2024-04-14T15:39:08.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: CherryPy Checker: 2024-04-14T15:39:08.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: The Application mounted at '' has an empty config. 2024-04-14T15:39:08.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:39:08.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:39:08] ENGINE Serving on http://:::9283 2024-04-14T15:39:08.841 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:08 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: [14/Apr/2024:15:39:08] ENGINE Bus STARTED 2024-04-14T15:39:09.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:08 smithi016 podman[91224]: 2024-04-14 15:39:08.947927518 +0000 UTC m=+0.621985938 container cleanup 152c4b9fda2092b8d247f1295f7f190fc009ae33dfb9121dca73fe144e2203f6 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:39:09.111 INFO:journalctl@ceph.mon.a.smithi016.stdout:Apr 14 15:39:08 smithi016 bash[91224]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-a 2024-04-14T15:39:09.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.a.service' 2024-04-14T15:39:09.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:39:09.476 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-14T15:39:09.476 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-14T15:39:09.476 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.c 2024-04-14T15:39:09.794 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:09 smithi016 systemd[1]: Stopping Ceph mon.c for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:39:10.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c[67344]: 2024-04-14T15:39:09.793+0000 7f37bfe41700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:39:10.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:09 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c[67344]: 2024-04-14T15:39:09.793+0000 7f37bfe41700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-04-14T15:39:10.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:09 smithi016 podman[91355]: 2024-04-14 15:39:09.907128522 +0000 UTC m=+0.202930284 container died da68d9a2587cb3f6d20190d467d264d33be530f3bd9652248f90850fa32eb278 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-14T15:39:10.546 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:10 smithi016 podman[91355]: 2024-04-14 15:39:10.295158205 +0000 UTC m=+0.590959975 container cleanup da68d9a2587cb3f6d20190d467d264d33be530f3bd9652248f90850fa32eb278 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-14T15:39:10.546 INFO:journalctl@ceph.mon.c.smithi016.stdout:Apr 14 15:39:10 smithi016 bash[91355]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-c 2024-04-14T15:39:10.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.c.service' 2024-04-14T15:39:10.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:39:10.706 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-14T15:39:10.706 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-14T15:39:10.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.b 2024-04-14T15:39:11.110 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:10 smithi196 systemd[1]: Stopping Ceph mon.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:39:11.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b[58675]: 2024-04-14T15:39:11.108+0000 7ff797fdb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:39:11.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b[58675]: 2024-04-14T15:39:11.108+0000 7ff797fdb700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-04-14T15:39:11.415 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:11 smithi196 podman[76785]: 2024-04-14 15:39:11.242475593 +0000 UTC m=+0.341854231 container died 72817f80f666c7de49b830bec27e1ed08cf6ec896a527e08b538ec376a141bf4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:39:11.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:11 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: ::ffff:172.21.15.196 - - [14/Apr/2024:15:39:11] "GET /metrics HTTP/1.1" 200 38771 "" "Prometheus/2.33.4" 2024-04-14T15:39:11.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:11 smithi196 podman[76785]: 2024-04-14 15:39:11.7010183 +0000 UTC m=+0.800396927 container cleanup 72817f80f666c7de49b830bec27e1ed08cf6ec896a527e08b538ec376a141bf4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-14T15:39:11.915 INFO:journalctl@ceph.mon.b.smithi196.stdout:Apr 14 15:39:11 smithi196 bash[76785]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mon-b 2024-04-14T15:39:12.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mon.b.service' 2024-04-14T15:39:12.166 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:39:12.166 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-14T15:39:12.166 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-14T15:39:12.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.y 2024-04-14T15:39:12.509 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:12 smithi016 systemd[1]: Stopping Ceph mgr.y for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:39:12.792 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:12 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:39:12.508+0000 7fda948a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:39:12.792 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:12 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y[60468]: 2024-04-14T15:39:12.508+0000 7fda948a0700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-14T15:39:12.793 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:12 smithi016 podman[91475]: 2024-04-14 15:39:12.540389083 +0000 UTC m=+0.176170062 container died e9559f3661299c2da10556bb2182e44f64dab37ac9bea69c17b6808464996dd9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403) 2024-04-14T15:39:13.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:12 smithi016 podman[91475]: 2024-04-14 15:39:12.875050423 +0000 UTC m=+0.510831406 container cleanup e9559f3661299c2da10556bb2182e44f64dab37ac9bea69c17b6808464996dd9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-a971fd5) 2024-04-14T15:39:13.111 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Apr 14 15:39:12 smithi016 bash[91475]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-y 2024-04-14T15:39:13.346 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.y.service' 2024-04-14T15:39:13.383 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:39:13.384 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-14T15:39:13.384 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-14T15:39:13.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.x 2024-04-14T15:39:13.665 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:39:13 smithi196 systemd[1]: Stopping Ceph mgr.x for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:39:14.007 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:39:13 smithi196 podman[76909]: 2024-04-14 15:39:13.74584245 +0000 UTC m=+0.155617535 container died c87a0109a594c505098fa34e957c65dcbc29e97395aa35289e516479b9ee6eff (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-14T15:39:14.300 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:39:14 smithi196 podman[76909]: 2024-04-14 15:39:14.089322685 +0000 UTC m=+0.499097762 container cleanup c87a0109a594c505098fa34e957c65dcbc29e97395aa35289e516479b9ee6eff (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.build-date=20240403, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:39:14.300 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Apr 14 15:39:14 smithi196 bash[76909]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-mgr-x 2024-04-14T15:39:14.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@mgr.x.service' 2024-04-14T15:39:14.540 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:39:14.541 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-14T15:39:14.541 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-14T15:39:14.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.0 2024-04-14T15:39:14.860 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:14 smithi016 systemd[1]: Stopping Ceph osd.0 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:39:15.361 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:14 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[70835]: 2024-04-14T15:39:14.864+0000 7f890df72700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:39:15.361 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:14 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[70835]: 2024-04-14T15:39:14.864+0000 7f890df72700 -1 osd.0 152 *** Got signal Terminated *** 2024-04-14T15:39:15.361 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:14 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0[70835]: 2024-04-14T15:39:14.864+0000 7f890df72700 -1 osd.0 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:39:20.425 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:20 smithi016 podman[91595]: 2024-04-14 15:39:20.173841354 +0000 UTC m=+5.437925132 container died f6ee7262903f6081e4ceb07846511851ebf40d52a7dbc809ad9aa00d95e31b57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-04-14T15:39:20.800 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:20 smithi016 podman[91595]: 2024-04-14 15:39:20.582368401 +0000 UTC m=+5.846452214 container cleanup f6ee7262903f6081e4ceb07846511851ebf40d52a7dbc809ad9aa00d95e31b57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-14T15:39:20.800 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:20 smithi016 bash[91595]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0 2024-04-14T15:39:21.110 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:20 smithi016 podman[91612]: 2024-04-14 15:39:20.800356915 +0000 UTC m=+0.619733432 container remove f6ee7262903f6081e4ceb07846511851ebf40d52a7dbc809ad9aa00d95e31b57 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:39:21.446 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:21 smithi016 podman[91705]: 2024-04-14 15:39:21.030814027 +0000 UTC m=+0.017692375 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:39:21.446 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:21 smithi016 podman[91705]: 2024-04-14 15:39:21.134829214 +0000 UTC m=+0.121707601 container create 951d855085d7c230f17c21850699e1cf5e8b6d4ca20cb018f213b362b5174745 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-14T15:39:21.446 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:21 smithi016 podman[91705]: 2024-04-14 15:39:21.363266665 +0000 UTC m=+0.350145026 container init 951d855085d7c230f17c21850699e1cf5e8b6d4ca20cb018f213b362b5174745 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:39:21.446 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:21 smithi016 podman[91705]: 2024-04-14 15:39:21.366657231 +0000 UTC m=+0.353535561 container start 951d855085d7c230f17c21850699e1cf5e8b6d4ca20cb018f213b362b5174745 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-04-14T15:39:21.447 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:21 smithi016 podman[91705]: 2024-04-14 15:39:21.446152584 +0000 UTC m=+0.433030934 container attach 951d855085d7c230f17c21850699e1cf5e8b6d4ca20cb018f213b362b5174745 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:39:21.861 INFO:journalctl@ceph.osd.0.smithi016.stdout:Apr 14 15:39:21 smithi016 podman[91705]: 2024-04-14 15:39:21.666144045 +0000 UTC m=+0.653022377 container died 951d855085d7c230f17c21850699e1cf5e8b6d4ca20cb018f213b362b5174745 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-0-deactivate, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-14T15:39:22.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.0.service' 2024-04-14T15:39:22.403 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:39:22.403 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-14T15:39:22.403 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-14T15:39:22.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.1 2024-04-14T15:39:22.611 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:22 smithi016 systemd[1]: Stopping Ceph osd.1 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:39:22.941 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[73180]: 2024-04-14T15:39:22.762+0000 7fe63061b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:39:22.941 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[73180]: 2024-04-14T15:39:22.762+0000 7fe63061b700 -1 osd.1 152 *** Got signal Terminated *** 2024-04-14T15:39:22.941 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:22 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1[73180]: 2024-04-14T15:39:22.762+0000 7fe63061b700 -1 osd.1 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:39:24.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:39:24 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:39:24.259Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-04-14T15:39:24.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:39:24 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:39:24.260Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-04-14T15:39:28.504 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:28 smithi016 podman[91811]: 2024-04-14 15:39:28.201168401 +0000 UTC m=+5.599376518 container stop 14dec0c76aaf52cc9a21ca47449b4cd27d9629f3d811d58f4095bbc971c6ae96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-14T15:39:28.504 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:28 smithi016 podman[91811]: 2024-04-14 15:39:28.231216304 +0000 UTC m=+5.629424467 container died 14dec0c76aaf52cc9a21ca47449b4cd27d9629f3d811d58f4095bbc971c6ae96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-14T15:39:28.788 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:28 smithi016 podman[91811]: 2024-04-14 15:39:28.603513804 +0000 UTC m=+6.001721930 container cleanup 14dec0c76aaf52cc9a21ca47449b4cd27d9629f3d811d58f4095bbc971c6ae96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-14T15:39:28.788 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:28 smithi016 bash[91811]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1 2024-04-14T15:39:29.110 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:28 smithi016 podman[91842]: 2024-04-14 15:39:28.788304284 +0000 UTC m=+0.580597940 container remove 14dec0c76aaf52cc9a21ca47449b4cd27d9629f3d811d58f4095bbc971c6ae96 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-14T15:39:29.492 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:29 smithi016 podman[91933]: 2024-04-14 15:39:29.13834791 +0000 UTC m=+0.017571651 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:39:29.492 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:29 smithi016 podman[91933]: 2024-04-14 15:39:29.264103081 +0000 UTC m=+0.143326821 container create 42c0d21e7accf8a528dc4d09daf475d44a212513cca7fcffa768e6cafad3df09 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403) 2024-04-14T15:39:29.796 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:29 smithi016 podman[91933]: 2024-04-14 15:39:29.492116163 +0000 UTC m=+0.371339913 container init 42c0d21e7accf8a528dc4d09daf475d44a212513cca7fcffa768e6cafad3df09 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-14T15:39:29.796 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:29 smithi016 podman[91933]: 2024-04-14 15:39:29.495683784 +0000 UTC m=+0.374907529 container start 42c0d21e7accf8a528dc4d09daf475d44a212513cca7fcffa768e6cafad3df09 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, RELEASE=main-a971fd5) 2024-04-14T15:39:29.796 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:29 smithi016 podman[91933]: 2024-04-14 15:39:29.575332709 +0000 UTC m=+0.454556452 container attach 42c0d21e7accf8a528dc4d09daf475d44a212513cca7fcffa768e6cafad3df09 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD) 2024-04-14T15:39:30.111 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:29 smithi016 podman[91933]: 2024-04-14 15:39:29.796757926 +0000 UTC m=+0.675981676 container died 42c0d21e7accf8a528dc4d09daf475d44a212513cca7fcffa768e6cafad3df09 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:39:30.410 INFO:journalctl@ceph.osd.1.smithi016.stdout:Apr 14 15:39:30 smithi016 podman[91959]: 2024-04-14 15:39:30.396176247 +0000 UTC m=+0.593444792 container remove 42c0d21e7accf8a528dc4d09daf475d44a212513cca7fcffa768e6cafad3df09 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-14T15:39:30.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.1.service' 2024-04-14T15:39:30.458 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:39:30.459 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-14T15:39:30.459 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-14T15:39:30.459 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.2 2024-04-14T15:39:30.791 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:30 smithi016 systemd[1]: Stopping Ceph osd.2 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:39:31.111 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[75657]: 2024-04-14T15:39:30.790+0000 7f3c7405b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:39:31.111 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[75657]: 2024-04-14T15:39:30.790+0000 7f3c7405b700 -1 osd.2 152 *** Got signal Terminated *** 2024-04-14T15:39:31.111 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:30 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2[75657]: 2024-04-14T15:39:30.790+0000 7f3c7405b700 -1 osd.2 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:39:34.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:39:34 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=error ts=2024-04-14T15:39:34.260Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-04-14T15:39:34.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:39:34 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:39:34.260Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-04-14T15:39:34.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:39:34 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:39:34.260Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-04-14T15:39:36.466 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:36 smithi016 podman[92038]: 2024-04-14 15:39:36.201748382 +0000 UTC m=+5.511180111 container died ce205c8399d32447883d67226dece43142090246c75e72b93839619d42a996fc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, io.buildah.version=1.33.5) 2024-04-14T15:39:36.733 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:36 smithi016 podman[92038]: 2024-04-14 15:39:36.548767969 +0000 UTC m=+5.858199699 container cleanup ce205c8399d32447883d67226dece43142090246c75e72b93839619d42a996fc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-14T15:39:36.733 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:36 smithi016 bash[92038]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2 2024-04-14T15:39:37.111 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:36 smithi016 podman[92056]: 2024-04-14 15:39:36.733351892 +0000 UTC m=+0.526985390 container remove ce205c8399d32447883d67226dece43142090246c75e72b93839619d42a996fc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403) 2024-04-14T15:39:37.504 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:37 smithi016 podman[92147]: 2024-04-14 15:39:37.069570942 +0000 UTC m=+0.017884027 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:39:37.504 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:37 smithi016 podman[92147]: 2024-04-14 15:39:37.175939564 +0000 UTC m=+0.124252698 container create 6b2b7d52995468664a415f35e3426baa36dcf831174fe87d521fbdcc3114c06f (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-deactivate, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True) 2024-04-14T15:39:37.504 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:37 smithi016 podman[92147]: 2024-04-14 15:39:37.504109119 +0000 UTC m=+0.452422196 container init 6b2b7d52995468664a415f35e3426baa36dcf831174fe87d521fbdcc3114c06f (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-deactivate, ceph=True, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:39:37.802 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:37 smithi016 podman[92147]: 2024-04-14 15:39:37.507687678 +0000 UTC m=+0.456000741 container start 6b2b7d52995468664a415f35e3426baa36dcf831174fe87d521fbdcc3114c06f (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-14T15:39:37.802 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:37 smithi016 podman[92147]: 2024-04-14 15:39:37.595609827 +0000 UTC m=+0.543922898 container attach 6b2b7d52995468664a415f35e3426baa36dcf831174fe87d521fbdcc3114c06f (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-a971fd5) 2024-04-14T15:39:38.082 INFO:journalctl@ceph.osd.2.smithi016.stdout:Apr 14 15:39:37 smithi016 podman[92147]: 2024-04-14 15:39:37.802113737 +0000 UTC m=+0.750426817 container died 6b2b7d52995468664a415f35e3426baa36dcf831174fe87d521fbdcc3114c06f (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5) 2024-04-14T15:39:38.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.2.service' 2024-04-14T15:39:38.409 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:39:38.409 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-14T15:39:38.409 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-14T15:39:38.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.3 2024-04-14T15:39:38.610 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:38 smithi016 systemd[1]: Stopping Ceph osd.3 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:39:39.111 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:38 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[77978]: 2024-04-14T15:39:38.735+0000 7f76354e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:39:39.111 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:38 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[77978]: 2024-04-14T15:39:38.735+0000 7f76354e0700 -1 osd.3 152 *** Got signal Terminated *** 2024-04-14T15:39:39.111 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:38 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[77978]: 2024-04-14T15:39:38.735+0000 7f76354e0700 -1 osd.3 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:39:40.861 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:40 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[77978]: 2024-04-14T15:39:40.539+0000 7f762dacc700 -1 osd.3 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:15.770055+0000 front 2024-04-14T15:39:15.770131+0000 (oldest deadline 2024-04-14T15:39:40.470193+0000) 2024-04-14T15:39:41.861 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:41 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[77978]: 2024-04-14T15:39:41.534+0000 7f762dacc700 -1 osd.3 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:15.770055+0000 front 2024-04-14T15:39:15.770131+0000 (oldest deadline 2024-04-14T15:39:40.470193+0000) 2024-04-14T15:39:41.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:41 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:41.561+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:42.665 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:42 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:42.522+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:42.665 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:42 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:42.551+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:42.861 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:42 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[77978]: 2024-04-14T15:39:42.561+0000 7f762dacc700 -1 osd.3 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:15.770055+0000 front 2024-04-14T15:39:15.770131+0000 (oldest deadline 2024-04-14T15:39:40.470193+0000) 2024-04-14T15:39:43.861 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:43 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3[77978]: 2024-04-14T15:39:43.597+0000 7f762dacc700 -1 osd.3 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:15.770055+0000 front 2024-04-14T15:39:15.770131+0000 (oldest deadline 2024-04-14T15:39:40.470193+0000) 2024-04-14T15:39:43.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:43.569+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:43.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:43 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:43.590+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:44.411 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:44 smithi016 podman[92251]: 2024-04-14 15:39:44.152127146 +0000 UTC m=+5.519859065 container died 6925155fa41a535cb5833ea796a0d9984e87b0cb4b80fe20ab03fde8e94c4761 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux ) 2024-04-14T15:39:44.411 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:39:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=error ts=2024-04-14T15:39:44.261Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-04-14T15:39:44.411 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:39:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:39:44.261Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-04-14T15:39:44.411 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:39:44 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:39:44.262Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-04-14T15:39:44.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:44 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:44.063+0000 7f3f2c4dd700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.793221+0000 front 2024-04-14T15:39:19.793355+0000 (oldest deadline 2024-04-14T15:39:43.293465+0000) 2024-04-14T15:39:44.778 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:44 smithi016 podman[92251]: 2024-04-14 15:39:44.519048021 +0000 UTC m=+5.886779933 container cleanup 6925155fa41a535cb5833ea796a0d9984e87b0cb4b80fe20ab03fde8e94c4761 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-14T15:39:44.778 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:44 smithi016 bash[92251]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3 2024-04-14T15:39:44.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:44 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:44.524+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:44.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:44 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:44.624+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:44.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:44 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:44.627+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:45.111 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:44 smithi016 podman[92271]: 2024-04-14 15:39:44.778459626 +0000 UTC m=+0.622505735 container remove 6925155fa41a535cb5833ea796a0d9984e87b0cb4b80fe20ab03fde8e94c4761 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3, org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-14T15:39:45.407 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:45 smithi016 podman[92365]: 2024-04-14 15:39:45.129276558 +0000 UTC m=+0.103121331 container create b158370d9f16ab36ba43dd2d5e8cc4ab6696098b8d044f72adeb66350e443af2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS) 2024-04-14T15:39:45.407 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:45 smithi016 podman[92365]: 2024-04-14 15:39:45.043912091 +0000 UTC m=+0.017756868 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:39:45.407 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:45 smithi016 podman[92365]: 2024-04-14 15:39:45.331838043 +0000 UTC m=+0.305682809 container init b158370d9f16ab36ba43dd2d5e8cc4ab6696098b8d044f72adeb66350e443af2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-14T15:39:45.407 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:45 smithi016 podman[92365]: 2024-04-14 15:39:45.334988772 +0000 UTC m=+0.308833545 container start b158370d9f16ab36ba43dd2d5e8cc4ab6696098b8d044f72adeb66350e443af2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5) 2024-04-14T15:39:45.414 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:45.068+0000 7f3f2c4dd700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.793221+0000 front 2024-04-14T15:39:19.793355+0000 (oldest deadline 2024-04-14T15:39:43.293465+0000) 2024-04-14T15:39:45.861 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:45 smithi016 podman[92365]: 2024-04-14 15:39:45.406874438 +0000 UTC m=+0.380719227 container attach b158370d9f16ab36ba43dd2d5e8cc4ab6696098b8d044f72adeb66350e443af2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-14T15:39:45.861 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:45 smithi016 podman[92365]: 2024-04-14 15:39:45.630609724 +0000 UTC m=+0.604454493 container died b158370d9f16ab36ba43dd2d5e8cc4ab6696098b8d044f72adeb66350e443af2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-04-14T15:39:45.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:45.607+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:45.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:45.503+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:45.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:45 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:45.631+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:46.164 INFO:journalctl@ceph.osd.3.smithi016.stdout:Apr 14 15:39:46 smithi016 podman[92391]: 2024-04-14 15:39:46.152268963 +0000 UTC m=+0.516977354 container remove b158370d9f16ab36ba43dd2d5e8cc4ab6696098b8d044f72adeb66350e443af2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.build-date=20240403, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-14T15:39:46.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.3.service' 2024-04-14T15:39:46.211 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:39:46.211 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-14T15:39:46.211 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-14T15:39:46.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.4 2024-04-14T15:39:46.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:46.057+0000 7f3f2c4dd700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.793221+0000 front 2024-04-14T15:39:19.793355+0000 (oldest deadline 2024-04-14T15:39:43.293465+0000) 2024-04-14T15:39:46.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:46 smithi196 systemd[1]: Stopping Ceph osd.4 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:39:46.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:46.529+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:46.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:46.590+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:46.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:46.592+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:46.916 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:46.506+0000 7f3f2fee9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:39:46.916 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:46.506+0000 7f3f2fee9700 -1 osd.4 152 *** Got signal Terminated *** 2024-04-14T15:39:46.916 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:46 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:46.506+0000 7f3f2fee9700 -1 osd.4 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:39:47.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:47.043+0000 7f3f2c4dd700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.793221+0000 front 2024-04-14T15:39:19.793355+0000 (oldest deadline 2024-04-14T15:39:43.293465+0000) 2024-04-14T15:39:47.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:47.586+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:47.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:47.532+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:47.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:47 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:47.566+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:48.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:48.085+0000 7f3f2c4dd700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.793221+0000 front 2024-04-14T15:39:19.793355+0000 (oldest deadline 2024-04-14T15:39:43.293465+0000) 2024-04-14T15:39:48.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:48.499+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:48.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:48.542+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:48.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:48 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:48.539+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:49.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:49 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:49.065+0000 7f3f2c4dd700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.793221+0000 front 2024-04-14T15:39:19.793355+0000 (oldest deadline 2024-04-14T15:39:43.293465+0000) 2024-04-14T15:39:49.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:49 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:49.065+0000 7f3f2c4dd700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.294917+0000 front 2024-04-14T15:39:27.294834+0000 (oldest deadline 2024-04-14T15:39:48.994852+0000) 2024-04-14T15:39:49.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:49 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:49.511+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:49.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:49 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:49.531+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:49.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:49 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:49.508+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:49.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:49 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:49.508+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:39:50.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:50 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:50.074+0000 7f3f2c4dd700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.793221+0000 front 2024-04-14T15:39:19.793355+0000 (oldest deadline 2024-04-14T15:39:43.293465+0000) 2024-04-14T15:39:50.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:50 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:50.074+0000 7f3f2c4dd700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.294917+0000 front 2024-04-14T15:39:27.294834+0000 (oldest deadline 2024-04-14T15:39:48.994852+0000) 2024-04-14T15:39:50.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:50 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:50.550+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:50.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:50 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:50.522+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:50.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:50 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:50.556+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:50.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:50 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:50.556+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:39:51.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:51 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:51.026+0000 7f3f2c4dd700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.793221+0000 front 2024-04-14T15:39:19.793355+0000 (oldest deadline 2024-04-14T15:39:43.293465+0000) 2024-04-14T15:39:51.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:51 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4[62335]: 2024-04-14T15:39:51.026+0000 7f3f2c4dd700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.294917+0000 front 2024-04-14T15:39:27.294834+0000 (oldest deadline 2024-04-14T15:39:48.994852+0000) 2024-04-14T15:39:51.820 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:51 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:51.522+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:51.820 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:51 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:51.522+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:39:51.820 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:51 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:51.554+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:51.820 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:51 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:51.554+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:26.810069+0000 front 2024-04-14T15:39:26.809982+0000 (oldest deadline 2024-04-14T15:39:51.410285+0000) 2024-04-14T15:39:51.821 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:51 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:51.564+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:51.821 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:51 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:51.564+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:39:52.123 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:51 smithi196 podman[77045]: 2024-04-14 15:39:51.820237841 +0000 UTC m=+5.411473044 container died 3a7864fa8601b80572c94dd3220e0b12a5afc98d68fe90e68488dbc5eae15331 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-14T15:39:52.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:52 smithi196 podman[77045]: 2024-04-14 15:39:52.289103023 +0000 UTC m=+5.880338233 container cleanup 3a7864fa8601b80572c94dd3220e0b12a5afc98d68fe90e68488dbc5eae15331 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS) 2024-04-14T15:39:52.415 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:52 smithi196 bash[77045]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4 2024-04-14T15:39:52.891 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:52 smithi196 podman[77062]: 2024-04-14 15:39:52.491013105 +0000 UTC m=+0.662030189 container remove 3a7864fa8601b80572c94dd3220e0b12a5afc98d68fe90e68488dbc5eae15331 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, io.buildah.version=1.33.5) 2024-04-14T15:39:52.892 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:52 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:52.488+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:52.892 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:52 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:52.488+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:39:52.892 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:52 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:52.556+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:52.892 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:52 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:52.556+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:26.810069+0000 front 2024-04-14T15:39:26.809982+0000 (oldest deadline 2024-04-14T15:39:51.410285+0000) 2024-04-14T15:39:52.893 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:52 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:52.581+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:52.893 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:52 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:52.581+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:39:53.165 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:52 smithi196 podman[77155]: 2024-04-14 15:39:52.891793173 +0000 UTC m=+0.110992506 container create 1fe488bb6389313e9228fdca4ce4ecba72756f7199ca14dbe5537556b6436d5a (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240403) 2024-04-14T15:39:53.165 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:52 smithi196 podman[77155]: 2024-04-14 15:39:52.798320311 +0000 UTC m=+0.017519645 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:39:53.165 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:53 smithi196 podman[77155]: 2024-04-14 15:39:53.127953935 +0000 UTC m=+0.347153286 container init 1fe488bb6389313e9228fdca4ce4ecba72756f7199ca14dbe5537556b6436d5a (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:39:53.165 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:53 smithi196 podman[77155]: 2024-04-14 15:39:53.131457404 +0000 UTC m=+0.350656745 container start 1fe488bb6389313e9228fdca4ce4ecba72756f7199ca14dbe5537556b6436d5a (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-14T15:39:53.424 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:53 smithi196 podman[77155]: 2024-04-14 15:39:53.219409151 +0000 UTC m=+0.438608495 container attach 1fe488bb6389313e9228fdca4ce4ecba72756f7199ca14dbe5537556b6436d5a (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-14T15:39:53.730 INFO:journalctl@ceph.osd.4.smithi196.stdout:Apr 14 15:39:53 smithi196 podman[77155]: 2024-04-14 15:39:53.425484221 +0000 UTC m=+0.644683558 container died 1fe488bb6389313e9228fdca4ce4ecba72756f7199ca14dbe5537556b6436d5a (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-14T15:39:53.730 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:53 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:53.473+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:53.731 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:53 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:53.473+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:39:53.731 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:53 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:53.516+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:53.731 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:53 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:53.516+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:26.810069+0000 front 2024-04-14T15:39:26.809982+0000 (oldest deadline 2024-04-14T15:39:51.410285+0000) 2024-04-14T15:39:53.731 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:53 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:53.600+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:53.731 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:53 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:53.600+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:39:54.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.4.service' 2024-04-14T15:39:54.093 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:39:54.093 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-14T15:39:54.093 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-14T15:39:54.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.5 2024-04-14T15:39:54.306 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:54 smithi196 systemd[1]: Stopping Ceph osd.5 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:39:54.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:39:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=error ts=2024-04-14T15:39:54.262Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-04-14T15:39:54.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:39:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:39:54.262Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-04-14T15:39:54.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:39:54 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:39:54.263Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-04-14T15:39:54.648 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:54.496+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:54.648 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:54.496+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:39:54.649 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:54.482+0000 7f4c9fa30700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:39:54.649 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:54.482+0000 7f4c9fa30700 -1 osd.5 152 *** Got signal Terminated *** 2024-04-14T15:39:54.649 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:54.482+0000 7f4c9fa30700 -1 osd.5 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:39:54.649 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:54.536+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:54.649 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:54.536+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:26.810069+0000 front 2024-04-14T15:39:26.809982+0000 (oldest deadline 2024-04-14T15:39:51.410285+0000) 2024-04-14T15:39:54.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:54.647+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:54.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:54 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:54.647+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:39:55.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:55 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:55.654+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:55.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:55 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:55.654+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:39:55.916 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:55 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:55.525+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:55.916 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:55 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:55.525+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:26.810069+0000 front 2024-04-14T15:39:26.809982+0000 (oldest deadline 2024-04-14T15:39:51.410285+0000) 2024-04-14T15:39:55.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:55 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:55.533+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:55.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:55 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:55.533+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:39:56.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:56.611+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:56.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:56.611+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:39:56.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:56.535+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:56.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:56.535+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:26.810069+0000 front 2024-04-14T15:39:26.809982+0000 (oldest deadline 2024-04-14T15:39:51.410285+0000) 2024-04-14T15:39:56.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:56.489+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:56.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:56 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:56.489+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:39:57.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:57.564+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:57.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:57.564+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:26.810069+0000 front 2024-04-14T15:39:26.809982+0000 (oldest deadline 2024-04-14T15:39:51.410285+0000) 2024-04-14T15:39:57.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:57.640+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:57.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:57.640+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:39:57.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:57.640+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.870307+0000 front 2024-04-14T15:39:34.870292+0000 (oldest deadline 2024-04-14T15:39:57.170181+0000) 2024-04-14T15:39:57.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:57.526+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:57.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:57 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:57.526+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:39:58.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:58.533+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:16.908367+0000 front 2024-04-14T15:39:16.908461+0000 (oldest deadline 2024-04-14T15:39:41.008289+0000) 2024-04-14T15:39:58.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:39:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5[64615]: 2024-04-14T15:39:58.533+0000 7f4c9c024700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:26.810069+0000 front 2024-04-14T15:39:26.809982+0000 (oldest deadline 2024-04-14T15:39:51.410285+0000) 2024-04-14T15:39:58.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:58.590+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:58.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:58.590+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:39:58.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:58.590+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.870307+0000 front 2024-04-14T15:39:34.870292+0000 (oldest deadline 2024-04-14T15:39:57.170181+0000) 2024-04-14T15:39:58.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:58.548+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:58.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:58 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:58.548+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:39:59.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:59.569+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:39:59.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:39:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:39:59.569+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:39:59.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:59.605+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:39:59.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:59.605+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:39:59.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:39:59 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:39:59.605+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.870307+0000 front 2024-04-14T15:39:34.870292+0000 (oldest deadline 2024-04-14T15:39:57.170181+0000) 2024-04-14T15:40:00.340 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:40:00 smithi196 podman[77261]: 2024-04-14 15:40:00.014440358 +0000 UTC m=+5.669178320 container died 08ddd21e09299f74a76e7de359297138040d9b5d2c8aa27fc51330220f9c27ed (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-14T15:40:00.618 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:40:00 smithi196 podman[77261]: 2024-04-14 15:40:00.548058336 +0000 UTC m=+6.202796337 container cleanup 08ddd21e09299f74a76e7de359297138040d9b5d2c8aa27fc51330220f9c27ed (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:40:00.619 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:40:00 smithi196 bash[77261]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5 2024-04-14T15:40:00.619 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:00 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:00.568+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:40:00.619 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:00 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:00.568+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:40:00.619 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:00 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:00.568+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.870307+0000 front 2024-04-14T15:39:34.870292+0000 (oldest deadline 2024-04-14T15:39:57.170181+0000) 2024-04-14T15:40:00.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:40:00 smithi196 podman[77289]: 2024-04-14 15:40:00.757957473 +0000 UTC m=+0.746554657 container remove 08ddd21e09299f74a76e7de359297138040d9b5d2c8aa27fc51330220f9c27ed (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0) 2024-04-14T15:40:00.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:00 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:00.617+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:00.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:00 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:00.617+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:00.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:00 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:00.617+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:01.485 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:40:01 smithi196 podman[77381]: 2024-04-14 15:40:01.166872146 +0000 UTC m=+0.113383897 container create a2fe47f8b86d4d63ffb0e1c4e87ea50e124fbb1d5dcaa100b488377e1c36ccca (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5) 2024-04-14T15:40:01.486 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:40:01 smithi196 podman[77381]: 2024-04-14 15:40:01.071467232 +0000 UTC m=+0.017978983 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:40:01.486 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:40:01 smithi196 podman[77381]: 2024-04-14 15:40:01.386117644 +0000 UTC m=+0.332629405 container init a2fe47f8b86d4d63ffb0e1c4e87ea50e124fbb1d5dcaa100b488377e1c36ccca (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-14T15:40:01.486 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:40:01 smithi196 podman[77381]: 2024-04-14 15:40:01.389411367 +0000 UTC m=+0.335923125 container start a2fe47f8b86d4d63ffb0e1c4e87ea50e124fbb1d5dcaa100b488377e1c36ccca (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-14T15:40:01.486 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:40:01 smithi196 podman[77381]: 2024-04-14 15:40:01.485863168 +0000 UTC m=+0.432374938 container attach a2fe47f8b86d4d63ffb0e1c4e87ea50e124fbb1d5dcaa100b488377e1c36ccca (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-deactivate, RELEASE=main-a971fd5, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403) 2024-04-14T15:40:01.915 INFO:journalctl@ceph.osd.5.smithi196.stdout:Apr 14 15:40:01 smithi196 podman[77381]: 2024-04-14 15:40:01.678984796 +0000 UTC m=+0.625496547 container died a2fe47f8b86d4d63ffb0e1c4e87ea50e124fbb1d5dcaa100b488377e1c36ccca (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:40:01.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:01.644+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:01.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:01.644+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:01.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:01.644+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:01.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:01.545+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:40:01.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:01.545+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:40:01.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:01 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:01.545+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.870307+0000 front 2024-04-14T15:39:34.870292+0000 (oldest deadline 2024-04-14T15:39:57.170181+0000) 2024-04-14T15:40:02.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.5.service' 2024-04-14T15:40:02.378 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:40:02.379 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-14T15:40:02.379 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-14T15:40:02.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.6 2024-04-14T15:40:02.641 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:02 smithi196 systemd[1]: Stopping Ceph osd.6 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:40:02.641 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:02.518+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:40:02.641 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:02.518+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:40:02.641 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:02.518+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.870307+0000 front 2024-04-14T15:39:34.870292+0000 (oldest deadline 2024-04-14T15:39:57.170181+0000) 2024-04-14T15:40:02.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:02.641+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:02.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:02.641+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:02.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:02.641+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:02.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:02.734+0000 7fe5d7cba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:40:02.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:02.734+0000 7fe5d7cba700 -1 osd.6 152 *** Got signal Terminated *** 2024-04-14T15:40:02.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:02 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:02.734+0000 7fe5d7cba700 -1 osd.6 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:40:03.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:03.516+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:40:03.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:03.516+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:40:03.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:03.516+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.870307+0000 front 2024-04-14T15:39:34.870292+0000 (oldest deadline 2024-04-14T15:39:57.170181+0000) 2024-04-14T15:40:03.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:03.610+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:03.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:03.610+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:03.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:03.610+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:03.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:03 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:03.610+0000 7efda81ea700 -1 osd.7 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.64400.0:34 2.8 2:10b29bcf:::realms.05a55308-7343-4eb3-a110-42efb7bb9b17.control:head [watch ping cookie 94429116201728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-04-14T15:40:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=error ts=2024-04-14T15:40:04.263Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-04-14T15:40:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:40:04.264Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-04-14T15:40:04.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:04 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:40:04.264Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-04-14T15:40:04.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:04.526+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:40:04.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:04.526+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:40:04.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:04.526+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.870307+0000 front 2024-04-14T15:39:34.870292+0000 (oldest deadline 2024-04-14T15:39:57.170181+0000) 2024-04-14T15:40:04.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:04.581+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:04.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:04.581+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:04.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:04.581+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:04.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:04.581+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:04.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:04 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:04.581+0000 7efda81ea700 -1 osd.7 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.64400.0:34 2.8 2:10b29bcf:::realms.05a55308-7343-4eb3-a110-42efb7bb9b17.control:head [watch ping cookie 94429116201728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-04-14T15:40:05.582 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:05.518+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:40:05.582 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:05.518+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:40:05.583 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:05.518+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.870307+0000 front 2024-04-14T15:39:34.870292+0000 (oldest deadline 2024-04-14T15:39:57.170181+0000) 2024-04-14T15:40:05.583 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:05.518+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:42.471098+0000 front 2024-04-14T15:39:42.471042+0000 (oldest deadline 2024-04-14T15:40:05.370961+0000) 2024-04-14T15:40:05.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:05.581+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:05.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:05.581+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:05.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:05.581+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:05.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:05.581+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:05.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:05 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:05.582+0000 7efda81ea700 -1 osd.7 152 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.64400.0:34 2.8 2:10b29bcf:::realms.05a55308-7343-4eb3-a110-42efb7bb9b17.control:head [watch ping cookie 94429116201728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152) 2024-04-14T15:40:06.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:06.626+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:06.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:06.626+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:06.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:06.626+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:06.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:06.626+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:06.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:06.565+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:40:06.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:06.565+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:40:06.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:06.565+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.870307+0000 front 2024-04-14T15:39:34.870292+0000 (oldest deadline 2024-04-14T15:39:57.170181+0000) 2024-04-14T15:40:06.916 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:06 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:06.565+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:42.471098+0000 front 2024-04-14T15:39:42.471042+0000 (oldest deadline 2024-04-14T15:40:05.370961+0000) 2024-04-14T15:40:07.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:07.583+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.366974+0000 front 2024-04-14T15:39:19.366858+0000 (oldest deadline 2024-04-14T15:39:42.266732+0000) 2024-04-14T15:40:07.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:07.583+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:27.368890+0000 front 2024-04-14T15:39:27.368746+0000 (oldest deadline 2024-04-14T15:39:50.868683+0000) 2024-04-14T15:40:07.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:07.583+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.870307+0000 front 2024-04-14T15:39:34.870292+0000 (oldest deadline 2024-04-14T15:39:57.170181+0000) 2024-04-14T15:40:07.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6[66885]: 2024-04-14T15:40:07.583+0000 7fe5d42ae700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:42.471098+0000 front 2024-04-14T15:39:42.471042+0000 (oldest deadline 2024-04-14T15:40:05.370961+0000) 2024-04-14T15:40:07.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:07.592+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:07.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:07.592+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:07.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:07.592+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:07.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:07 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:07.592+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:08.318 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:08 smithi196 podman[77485]: 2024-04-14 15:40:08.057440348 +0000 UTC m=+5.457715373 container died 31fa3bea2859b81804bd4334af7826b6a26b6fda6ae86dbdba25b56ac4588622 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:40:08.665 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:08 smithi196 podman[77485]: 2024-04-14 15:40:08.475878831 +0000 UTC m=+5.876153852 container cleanup 31fa3bea2859b81804bd4334af7826b6a26b6fda6ae86dbdba25b56ac4588622 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, ceph=True) 2024-04-14T15:40:08.665 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:08 smithi196 bash[77485]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6 2024-04-14T15:40:08.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:08.561+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:08.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:08.561+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:08.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:08.561+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:08.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:08 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:08.561+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:09.139 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:08 smithi196 podman[77503]: 2024-04-14 15:40:08.719751581 +0000 UTC m=+0.658561552 container remove 31fa3bea2859b81804bd4334af7826b6a26b6fda6ae86dbdba25b56ac4588622 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-14T15:40:09.415 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:09 smithi196 podman[77598]: 2024-04-14 15:40:09.04211473 +0000 UTC m=+0.017555317 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:40:09.415 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:09 smithi196 podman[77598]: 2024-04-14 15:40:09.153014939 +0000 UTC m=+0.128455497 container create 2ce9a6f2a2ada1d6f61d8d5540142a21596cb17d07c2c49cde156a93ef9a4ef7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-14T15:40:09.415 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:09 smithi196 podman[77598]: 2024-04-14 15:40:09.372511146 +0000 UTC m=+0.347951713 container init 2ce9a6f2a2ada1d6f61d8d5540142a21596cb17d07c2c49cde156a93ef9a4ef7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-14T15:40:09.415 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:09 smithi196 podman[77598]: 2024-04-14 15:40:09.375918103 +0000 UTC m=+0.351358665 container start 2ce9a6f2a2ada1d6f61d8d5540142a21596cb17d07c2c49cde156a93ef9a4ef7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T15:40:09.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:09 smithi196 podman[77598]: 2024-04-14 15:40:09.472265715 +0000 UTC m=+0.447706302 container attach 2ce9a6f2a2ada1d6f61d8d5540142a21596cb17d07c2c49cde156a93ef9a4ef7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-04-14T15:40:09.915 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:09 smithi196 podman[77598]: 2024-04-14 15:40:09.663135743 +0000 UTC m=+0.638576297 container died 2ce9a6f2a2ada1d6f61d8d5540142a21596cb17d07c2c49cde156a93ef9a4ef7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T15:40:09.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:09.546+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:09.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:09.546+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:09.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:09.546+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:09.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:09 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:09.546+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:10.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.6.service' 2024-04-14T15:40:10.216 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:10 smithi196 podman[77625]: 2024-04-14 15:40:10.175171813 +0000 UTC m=+0.502034224 container remove 2ce9a6f2a2ada1d6f61d8d5540142a21596cb17d07c2c49cde156a93ef9a4ef7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-14T15:40:10.216 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:10 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.6.service: Deactivated successfully. 2024-04-14T15:40:10.216 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:10 smithi196 systemd[1]: Stopped Ceph osd.6 for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:40:10.216 INFO:journalctl@ceph.osd.6.smithi196.stdout:Apr 14 15:40:10 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.6.service: Consumed 4.311s CPU time. 2024-04-14T15:40:10.231 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:40:10.231 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-14T15:40:10.231 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-14T15:40:10.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.7 2024-04-14T15:40:10.553 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:10 smithi196 systemd[1]: Stopping Ceph osd.7 for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:40:10.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:10 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:10.552+0000 7efdafbfe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T15:40:10.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:10 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:10.552+0000 7efdafbfe700 -1 osd.7 152 *** Got signal Terminated *** 2024-04-14T15:40:10.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:10 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:10.552+0000 7efdafbfe700 -1 osd.7 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T15:40:10.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:10 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:10.562+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:10.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:10 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:10.562+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:10.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:10 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:10.562+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:10.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:10 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:10.562+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:11.817 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:11.552+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:11.817 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:11.552+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:11.817 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:11.552+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:11.817 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:11.552+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:11.818 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:11.782Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-04-14T15:40:11.818 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:11.786Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-04-14T15:40:11.818 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:11.786Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-04-14T15:40:11.818 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:11.787Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-04-14T15:40:11.818 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:11 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:11.787Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-04-14T15:40:12.830 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:12.519+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:12.831 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:12.519+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:12.831 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:12.519+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:12.831 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:12.519+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:12.831 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:12 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:12.519+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-04-14T15:39:50.559840+0000 front 2024-04-14T15:39:50.559871+0000 (oldest deadline 2024-04-14T15:40:12.260036+0000) 2024-04-14T15:40:13.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:13.486+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:13.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:13.486+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:13.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:13.486+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:13.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:13.486+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:13.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:13 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:13.486+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-04-14T15:39:50.559840+0000 front 2024-04-14T15:39:50.559871+0000 (oldest deadline 2024-04-14T15:40:12.260036+0000) 2024-04-14T15:40:14.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:14 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=error ts=2024-04-14T15:40:14.264Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-04-14T15:40:14.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:14 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:40:14.264Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi016.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.16:8443: connect: connection refused" 2024-04-14T15:40:14.611 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:14 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=warn ts=2024-04-14T15:40:14.265Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi196.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.196:8443: connect: connection refused" 2024-04-14T15:40:14.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:14 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:14.496+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:14.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:14 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:14.496+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:14.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:14 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:14.496+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:14.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:14 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:14.496+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:14.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:14 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:14.496+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-04-14T15:39:50.559840+0000 front 2024-04-14T15:39:50.559871+0000 (oldest deadline 2024-04-14T15:40:12.260036+0000) 2024-04-14T15:40:15.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:15 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:15.531+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-04-14T15:39:19.855560+0000 front 2024-04-14T15:39:19.855667+0000 (oldest deadline 2024-04-14T15:39:43.955357+0000) 2024-04-14T15:40:15.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:15 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:15.531+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-04-14T15:39:25.656446+0000 front 2024-04-14T15:39:25.656331+0000 (oldest deadline 2024-04-14T15:39:49.156222+0000) 2024-04-14T15:40:15.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:15 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:15.531+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-04-14T15:39:34.957544+0000 front 2024-04-14T15:39:34.957600+0000 (oldest deadline 2024-04-14T15:40:00.257418+0000) 2024-04-14T15:40:15.915 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:15 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:15.531+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.16:6830 osd.3 since back 2024-04-14T15:39:43.658575+0000 front 2024-04-14T15:39:43.658549+0000 (oldest deadline 2024-04-14T15:40:04.158530+0000) 2024-04-14T15:40:15.916 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:15 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7[69174]: 2024-04-14T15:40:15.531+0000 7efda81ea700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-04-14T15:39:50.559840+0000 front 2024-04-14T15:39:50.559871+0000 (oldest deadline 2024-04-14T15:40:12.260036+0000) 2024-04-14T15:40:16.312 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:16 smithi196 podman[77703]: 2024-04-14 15:40:16.005318438 +0000 UTC m=+5.558066881 container died 1cf6aa633b3f94a239831a5ece70283908dc5062ce59c66eb7bd81924c040ee8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-14T15:40:16.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:16 smithi196 podman[77703]: 2024-04-14 15:40:16.502629607 +0000 UTC m=+6.055378040 container cleanup 1cf6aa633b3f94a239831a5ece70283908dc5062ce59c66eb7bd81924c040ee8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-14T15:40:16.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:16 smithi196 bash[77703]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7 2024-04-14T15:40:17.061 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:16 smithi196 podman[77732]: 2024-04-14 15:40:16.762230391 +0000 UTC m=+0.757949053 container remove 1cf6aa633b3f94a239831a5ece70283908dc5062ce59c66eb7bd81924c040ee8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:40:17.340 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:17 smithi196 podman[77829]: 2024-04-14 15:40:16.964060132 +0000 UTC m=+0.017802161 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T15:40:17.340 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:17 smithi196 podman[77829]: 2024-04-14 15:40:17.062976443 +0000 UTC m=+0.116718457 container create d87fc66aeb11cac5c4e3d2dd765dff2c0c2c6c3f2a8c8ab4dccf95daeac71b73 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-14T15:40:17.340 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:17 smithi196 podman[77829]: 2024-04-14 15:40:17.265462289 +0000 UTC m=+0.319204301 container init d87fc66aeb11cac5c4e3d2dd765dff2c0c2c6c3f2a8c8ab4dccf95daeac71b73 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-deactivate, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, io.buildah.version=1.33.5) 2024-04-14T15:40:17.340 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:17 smithi196 podman[77829]: 2024-04-14 15:40:17.268468463 +0000 UTC m=+0.322210481 container start d87fc66aeb11cac5c4e3d2dd765dff2c0c2c6c3f2a8c8ab4dccf95daeac71b73 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T15:40:17.340 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:17 smithi196 podman[77829]: 2024-04-14 15:40:17.340314532 +0000 UTC m=+0.394056552 container attach d87fc66aeb11cac5c4e3d2dd765dff2c0c2c6c3f2a8c8ab4dccf95daeac71b73 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-a971fd5) 2024-04-14T15:40:17.665 INFO:journalctl@ceph.osd.7.smithi196.stdout:Apr 14 15:40:17 smithi196 podman[77829]: 2024-04-14 15:40:17.561921868 +0000 UTC m=+0.615663888 container died d87fc66aeb11cac5c4e3d2dd765dff2c0c2c6c3f2a8c8ab4dccf95daeac71b73 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403) 2024-04-14T15:40:18.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@osd.7.service' 2024-04-14T15:40:18.168 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:40:18.168 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-14T15:40:18.169 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-14T15:40:18.169 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@prometheus.a 2024-04-14T15:40:18.363 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 systemd[1]: Stopping Ceph prometheus.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:40:18.622 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T15:40:18.622 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-14T15:40:18.622 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-14T15:40:18.622 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-14T15:40:18.622 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-14T15:40:18.622 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-14T15:40:18.622 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T15:40:18.622 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-14T15:40:18.622 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T15:40:18.622 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T15:40:18.623 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-14T15:40:18.623 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a[54068]: ts=2024-04-14T15:40:18.363Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-14T15:40:18.623 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 podman[77908]: 2024-04-14 15:40:18.373719932 +0000 UTC m=+0.114658279 container died 9f60db35d8a82abdbf6712da25a17e32c1047b2e8da6e2f7ea94d11b65707945 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:40:18.872 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 podman[77908]: 2024-04-14 15:40:18.712589593 +0000 UTC m=+0.453527945 container cleanup 9f60db35d8a82abdbf6712da25a17e32c1047b2e8da6e2f7ea94d11b65707945 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T15:40:18.872 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Apr 14 15:40:18 smithi196 bash[77908]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T15:40:18.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@prometheus.a.service' 2024-04-14T15:40:18.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T15:40:18.980 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-14T15:40:18.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 --force --keep-logs 2024-04-14T15:40:20.861 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:40:20 smithi016 systemd[1]: Stopping Ceph node-exporter.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:40:20.861 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:40:20 smithi016 podman[92732]: 2024-04-14 15:40:20.60581608 +0000 UTC m=+0.107004240 container died cb8e94cfa1ed06f3afe4dc1379319dffad72cf87e31522983deb24334bd67835 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:40:21.237 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:40:20 smithi016 podman[92732]: 2024-04-14 15:40:20.961660223 +0000 UTC m=+0.462848384 container cleanup cb8e94cfa1ed06f3afe4dc1379319dffad72cf87e31522983deb24334bd67835 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:40:21.238 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:40:20 smithi016 bash[92732]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a 2024-04-14T15:40:21.238 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:40:20 smithi016 bash[92760]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter.a" found: no such container 2024-04-14T15:40:21.524 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:21 smithi016 systemd[1]: Stopping Ceph alertmanager.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:40:21.525 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:40:21 smithi016 podman[92744]: 2024-04-14 15:40:21.237914684 +0000 UTC m=+0.632286187 container remove cb8e94cfa1ed06f3afe4dc1379319dffad72cf87e31522983deb24334bd67835 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T15:40:21.525 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:40:21 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-14T15:40:21.525 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:40:21 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-14T15:40:21.525 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:40:21 smithi016 systemd[1]: Stopped Ceph node-exporter.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:40:21.525 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Apr 14 15:40:21 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.a.service: Consumed 3.613s CPU time. 2024-04-14T15:40:21.809 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:21 smithi016 ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a[59204]: level=info ts=2024-04-14T15:40:21.524Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-14T15:40:21.809 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:21 smithi016 podman[92813]: 2024-04-14 15:40:21.575662354 +0000 UTC m=+0.145362295 container died a29a63fa714603b4bde3bda643d2e3f72b15b8df08cb939f4ce3a81c1b65d4a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:40:22.101 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:21 smithi016 podman[92813]: 2024-04-14 15:40:21.900390895 +0000 UTC m=+0.470090836 container cleanup a29a63fa714603b4bde3bda643d2e3f72b15b8df08cb939f4ce3a81c1b65d4a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:40:22.101 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:21 smithi016 bash[92813]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a 2024-04-14T15:40:22.360 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:22 smithi016 podman[92827]: 2024-04-14 15:40:22.101371245 +0000 UTC m=+0.530953520 container remove a29a63fa714603b4bde3bda643d2e3f72b15b8df08cb939f4ce3a81c1b65d4a0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T15:40:22.361 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:22 smithi016 podman[92827]: 2024-04-14 15:40:22.177186284 +0000 UTC m=+0.606768562 volume remove a68c846189e240045ee5fc2ecbaffbc306b0d084f33f5745d9fff0e4949e145f 2024-04-14T15:40:22.361 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:22 smithi016 bash[92839]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-alertmanager.a" found: no such container 2024-04-14T15:40:22.361 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:22 smithi016 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-14T15:40:22.361 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Apr 14 15:40:22 smithi016 systemd[1]: Stopped Ceph alertmanager.a for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:40:56.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 --force --keep-logs 2024-04-14T15:40:58.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:40:57 smithi196 systemd[1]: Stopping Ceph node-exporter.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968... 2024-04-14T15:40:58.165 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:40:57 smithi196 podman[78266]: 2024-04-14 15:40:57.98821678 +0000 UTC m=+0.115535226 container died bea8d0b8e46e0751891ed3f036e1e23b2f7660cce66d36cf6311596e97f87a0a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:40:58.432 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:40:58 smithi196 podman[78266]: 2024-04-14 15:40:58.402696947 +0000 UTC m=+0.530015392 container cleanup bea8d0b8e46e0751891ed3f036e1e23b2f7660cce66d36cf6311596e97f87a0a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:40:58.432 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:40:58 smithi196 bash[78266]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b 2024-04-14T15:40:58.810 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:40:58 smithi196 bash[78293]: Error: no container with name or ID "ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter.b" found: no such container 2024-04-14T15:40:58.811 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:40:58 smithi196 podman[78280]: 2024-04-14 15:40:58.620842165 +0000 UTC m=+0.635116090 container remove bea8d0b8e46e0751891ed3f036e1e23b2f7660cce66d36cf6311596e97f87a0a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T15:40:58.811 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:40:58 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-14T15:40:58.811 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:40:58 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-14T15:40:58.811 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:40:58 smithi196 systemd[1]: Stopped Ceph node-exporter.b for 3ca5917a-fa72-11ee-bc8e-c7b262605968. 2024-04-14T15:40:58.811 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Apr 14 15:40:58 smithi196 systemd[1]: ceph-3ca5917a-fa72-11ee-bc8e-c7b262605968@node-exporter.b.service: Consumed 3.568s CPU time. 2024-04-14T15:41:21.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:41:21.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T15:41:21.815 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-14T15:41:21.817 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101/remote/smithi016/crash 2024-04-14T15:41:21.818 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/crash -- . 2024-04-14T15:41:21.871 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/crash: Cannot open: No such file or directory 2024-04-14T15:41:21.871 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-04-14T15:41:21.873 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101/remote/smithi196/crash 2024-04-14T15:41:21.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/crash -- . 2024-04-14T15:41:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:tar: /var/lib/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/crash: Cannot open: No such file or directory 2024-04-14T15:41:21.911 INFO:teuthology.orchestra.run.smithi196.stderr:tar: Error is not recoverable: exiting now 2024-04-14T15:41:21.912 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-14T15:41:21.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T15:41:21.941 INFO:teuthology.orchestra.run.smithi016.stdout:2024-04-14T15:20:31.746025+0000 mon.a (mon.0) 167 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-14T15:41:21.943 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-14T15:41:21.943 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[SEC\]' /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T15:41:22.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]' /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T15:41:22.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[WRN\]' /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T15:41:22.151 INFO:teuthology.orchestra.run.smithi016.stdout:2024-04-14T15:20:31.746025+0000 mon.a (mon.0) 167 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-14T15:41:22.153 INFO:tasks.cephadm:Compressing logs... 2024-04-14T15:41:22.153 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-14T15:41:22.196 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-14T15:41:22.220 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-14T15:41:22.220 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-14T15:41:22.220 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mon.a.log 2024-04-14T15:41:22.220 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.audit.log 2024-04-14T15:41:22.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cephadm.log: gzip/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.smpl.smithi016.sjsxjs.log 2024-04-14T15:41:22.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.1.log 2024-04-14T15:41:22.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.smpl.smithi016.sjsxjs.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.3.log 2024-04-14T15:41:22.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.1.log: 76.5% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.smpl.smithi016.sjsxjs.log.gz 2024-04-14T15:41:22.222 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mgr.y.log 2024-04-14T15:41:22.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi016.qozpor.log 2024-04-14T15:41:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/tcmu-runner.log 2024-04-14T15:41:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi016.qozpor.log: 76.5% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi016.qozpor.log.gz 2024-04-14T15:41:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.0.log 2024-04-14T15:41:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/tcmu-runner.log: 84.4% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/tcmu-runner.log.gz 2024-04-14T15:41:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.cephadm.log 2024-04-14T15:41:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.0.log: 94.3% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.audit.log.gz 2024-04-14T15:41:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.log 2024-04-14T15:41:22.230 INFO:teuthology.orchestra.run.smithi196.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:: No such file or directory 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.6.log 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.audit.log 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.4.log 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.7.log 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.5.log 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mgr.x.log 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.smpl.smithi196.uwpszz.log 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.cephadm.log 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.smpl.smithi196.uwpszz.log: 76.5% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.smpl.smithi196.uwpszz.log.gz 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.log 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.cephadm.log: 85.8% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.cephadm.log.gz 2024-04-14T15:41:22.231 INFO:teuthology.orchestra.run.smithi196.stderr: 90.6% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.audit.log.gz 2024-04-14T15:41:22.232 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-volume.log 2024-04-14T15:41:22.232 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mon.b.log 2024-04-14T15:41:22.233 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-volume.log: 87.7% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.log.gz 2024-04-14T15:41:22.233 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.cephadm.log: 91.8% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.cephadm.log.gz 2024-04-14T15:41:22.233 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.2.log 2024-04-14T15:41:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.log: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-14T15:41:22.235 INFO:teuthology.orchestra.run.smithi016.stderr: 93.1% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph.log.gz 2024-04-14T15:41:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-volume.log 2024-04-14T15:41:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mon.c.log 2024-04-14T15:41:22.243 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi196.wjbrhf.log 2024-04-14T15:41:22.243 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mon.b.log: 91.4% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-volume.log.gz 2024-04-14T15:41:22.244 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi196.wjbrhf.log: 90.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-14T15:41:22.244 INFO:teuthology.orchestra.run.smithi196.stderr: 76.5% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi196.wjbrhf.log.gz 2024-04-14T15:41:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-volume.log: /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mon.c.log: 91.4% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-volume.log.gz 2024-04-14T15:41:22.277 INFO:teuthology.orchestra.run.smithi196.stderr: 89.7% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mgr.x.log.gz 2024-04-14T15:41:22.537 INFO:teuthology.orchestra.run.smithi196.stderr: 92.4% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mon.b.log.gz 2024-04-14T15:41:22.563 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mgr.y.log.gz 2024-04-14T15:41:22.743 INFO:teuthology.orchestra.run.smithi016.stderr: 92.2% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mon.c.log.gz 2024-04-14T15:41:23.180 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-mon.a.log.gz 2024-04-14T15:41:23.311 INFO:teuthology.orchestra.run.smithi016.stderr: 93.6% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.2.log.gz 2024-04-14T15:41:23.317 INFO:teuthology.orchestra.run.smithi196.stderr: 93.5% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.6.log.gz 2024-04-14T15:41:23.559 INFO:teuthology.orchestra.run.smithi196.stderr: 93.8% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.7.log.gz 2024-04-14T15:41:23.656 INFO:teuthology.orchestra.run.smithi196.stderr: 93.6% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.5.log.gz 2024-04-14T15:41:23.662 INFO:teuthology.orchestra.run.smithi196.stderr: 93.7% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.4.log.gz 2024-04-14T15:41:23.664 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-04-14T15:41:23.664 INFO:teuthology.orchestra.run.smithi196.stderr:real 0m1.454s 2024-04-14T15:41:23.664 INFO:teuthology.orchestra.run.smithi196.stderr:user 0m5.451s 2024-04-14T15:41:23.664 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m0.260s 2024-04-14T15:41:23.683 INFO:teuthology.orchestra.run.smithi016.stderr: 93.6% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.0.log.gz 2024-04-14T15:41:23.767 INFO:teuthology.orchestra.run.smithi016.stderr: 93.6% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.1.log.gz 2024-04-14T15:41:24.016 INFO:teuthology.orchestra.run.smithi016.stderr: 93.6% -- replaced with /var/log/ceph/3ca5917a-fa72-11ee-bc8e-c7b262605968/ceph-osd.3.log.gz 2024-04-14T15:41:24.018 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-14T15:41:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m1.811s 2024-04-14T15:41:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m7.406s 2024-04-14T15:41:24.019 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m0.275s 2024-04-14T15:41:24.019 INFO:tasks.cephadm:Archiving logs... 2024-04-14T15:41:24.020 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101/remote/smithi016/log 2024-04-14T15:41:24.021 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-14T15:41:25.063 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101/remote/smithi196/log 2024-04-14T15:41:25.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-14T15:41:25.933 INFO:tasks.cephadm:Removing cluster... 2024-04-14T15:41:25.934 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 --force 2024-04-14T15:41:26.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3ca5917a-fa72-11ee-bc8e-c7b262605968 --force 2024-04-14T15:41:26.522 INFO:tasks.cephadm:Removing cephadm ... 2024-04-14T15:41:26.522 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-14T15:41:26.537 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-14T15:41:26.553 INFO:tasks.cephadm:Teardown complete 2024-04-14T15:41:26.554 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-14T15:41:26.563 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-14T15:41:26.563 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T15:41:26.579 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T15:41:26.591 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-14T15:41:26.595 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T15:41:26.595 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-14T15:41:26.595 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv01.front.sepia.ceph.com 3 7 377 15 +1169us[+1170us] +/- 63ms 2024-04-14T15:41:26.595 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 16 -1606us[-1605us] +/- 52ms 2024-04-14T15:41:26.595 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +515us[ +516us] +/- 65ms 2024-04-14T15:41:26.595 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:41:26.606 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-04-14T15:41:26.610 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T15:41:26.610 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-04-14T15:41:26.610 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv01.front.sepia.ceph.com 3 6 377 17 +1231us[+1237us] +/- 63ms 2024-04-14T15:41:26.611 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 18 -1589us[-1583us] +/- 52ms 2024-04-14T15:41:26.611 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 +576us[ +576us] +/- 65ms 2024-04-14T15:41:26.611 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T15:41:26.611 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-14T15:41:26.622 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-14T15:41:26.623 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-14T15:41:26.631 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-14T15:41:26.719 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-14T15:41:26.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-14T15:41:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:41:26.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-14T15:41:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:41:26.857 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-14T15:41:26.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-14T15:41:26.876 INFO:teuthology.task.internal:Duration was 1643.407265 seconds 2024-04-14T15:41:26.876 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-14T15:41:26.898 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-14T15:41:26.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-14T15:41:26.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-14T15:41:26.942 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T15:41:26.943 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T15:41:27.394 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-14T15:41:27.394 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-04-14T15:41:27.394 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-14T15:41:27.455 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-04-14T15:41:27.455 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-14T15:41:27.483 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-14T15:41:27.484 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-14T15:41:27.498 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-14T15:41:27.557 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-14T15:41:27.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-14T15:41:27.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-14T15:41:27.993 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-14T15:41:28.004 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-14T15:41:28.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-14T15:41:28.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-14T15:41:28.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-14T15:41:28.068 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-14T15:41:28.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-14T15:41:28.096 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-04-14T15:41:28.123 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-04-14T15:41:28.152 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-14T15:41:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:41:28.177 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-14T15:41:28.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T15:41:28.206 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-14T15:41:28.216 INFO:teuthology.task.internal:Transferring archived files... 2024-04-14T15:41:28.217 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101/remote/smithi016 2024-04-14T15:41:28.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-14T15:41:28.267 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101/remote/smithi196 2024-04-14T15:41:28.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-14T15:41:28.309 INFO:teuthology.task.internal:Removing archive directory... 2024-04-14T15:41:28.309 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-14T15:41:28.311 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-14T15:41:28.356 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-14T15:41:28.368 INFO:teuthology.task.internal:Not uploading archives. 2024-04-14T15:41:28.368 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-14T15:41:28.378 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-14T15:41:28.378 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-14T15:41:28.381 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-14T15:41:28.392 INFO:teuthology.orchestra.run.smithi016.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 14 15:41 /home/ubuntu/cephtest 2024-04-14T15:41:28.410 INFO:teuthology.orchestra.run.smithi196.stdout: 265793 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 14 15:41 /home/ubuntu/cephtest 2024-04-14T15:41:28.412 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-14T15:41:28.422 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-14T15:41:28.443 INFO:teuthology.nuke:Checking targets against current locks 2024-04-14T15:41:28.472 DEBUG:teuthology.nuke:shortname: smithi016 2024-04-14T15:41:28.473 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T15:41:28.496 DEBUG:teuthology.nuke:shortname: smithi196 2024-04-14T15:41:28.496 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T15:41:28.499 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 15:05:17.462356', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErCE7+8gl7quRSCjfzY/p94d7HgNCvq0YDZuLdYe2oBhin9jt1ZuEoE37DUQp+Q/4noVNS86OGPA+Ci+98CCe0='} 2024-04-14T15:41:28.517 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 15:05:17.464368', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJH71klPmbDk9qxu6LD4ThsEk70DrFl7zWxBL+z/+uJV+gTUIyU1CkXRPtjFDXa4F7Rdc7vmCftFkyueFlT9uzI='} 2024-04-14T15:41:28.545 INFO:teuthology.orchestra.console.smithi016:Power off 2024-04-14T15:41:28.545 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-14T15:41:28.570 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2024-04-14T15:41:28.570 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:41:28.586 INFO:teuthology.orchestra.console.smithi196:Power off 2024-04-14T15:41:28.586 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-14T15:41:28.595 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-04-14T15:41:28.608 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-04-14T15:41:28.608 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:41:28.631 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-04-14T15:41:32.597 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:41:32.632 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:41:32.706 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-04-14T15:41:32.740 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-04-14T15:41:36.711 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:41:36.742 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:41:36.822 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2024-04-14T15:41:36.850 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-04-14T15:41:36.923 INFO:teuthology.orchestra.console.smithi016:Power off completed 2024-04-14T15:41:40.854 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T15:41:40.965 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-04-14T15:41:41.066 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-04-14T15:41:41.171 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} duration: 1643.4072654247284 failure_reason: '"2024-04-14T15:20:31.746025+0000 mon.a (mon.0) 167 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-04-14T15:41:41.172 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-14T15:41:41.260 INFO:teuthology.run:FAIL